Author: stevel
Date: Sat Dec 19 16:51:51 2015
New Revision: 1720946
URL: http://svn.apache.org/viewvc?rev=1720946&view=rev
Log:
SLIDER-1014 current doc more a log of failed attempt than a release runbook
Modified:
incubator/slider/site/trunk/content/developing/releasing.md
Modified: incubator/slider/site/trunk/content/developing/releasing.md
URL:
http://svn.apache.org/viewvc/incubator/slider/site/trunk/content/developing/releasing.md?rev=1720946&r1=1720945&r2=1720946&view=diff
==============================================================================
--- incubator/slider/site/trunk/content/developing/releasing.md (original)
+++ incubator/slider/site/trunk/content/developing/releasing.md Sat Dec 19
16:51:51 2015
@@ -37,8 +37,15 @@ git graph.
Read the [ASF incubator release
manual](http://incubator.apache.org/guides/releasemanagement.html)
+Make sure that you are building with the JDK of the minimum supported version
+for Slider. Code compiled with later major versions of Java don't work
+on earlier versions.
+
+Check [the Jenkins builds](https://builds.apache.org/job/Slider-develop/) to
+verify that the build is stable.
+
Check out the latest version of the branch to be released,
-run the tests. This should be done on a checked out
+run the tests. You should release a checked out
version of the code that is not the one you are developing on
(ideally, a clean VM), to ensure that you aren't releasing a slightly
modified version of your own, and that you haven't accidentally
@@ -54,6 +61,14 @@ create HBase and Accumulo clusters in th
*Make sure that the integration tests are passing (and not being skipped)
before
starting to make a release*
+A good test matrix is:
+
+* Insecure, Secure
+* Linux, Windows
+* Java 7, Java 8
+
+A pair of VMs are sufficient to cover this.
+
Make sure there are no uncommitted files in your local repo.
If you are not building against a stable Hadoop release
@@ -68,14 +83,17 @@ Unstable Hadoop versions MUST NOT be use
Slider release.
-
*5.* Check ASF headers dependencies
Ensure the rat check is clean. This rejects files which lack ASF headers
- mvn clean
- mvn apache-rat:check -Prat
+```
+mvn clean
+mvn apache-rat:check -Prat
+```
+(if you've got packages that aren't linked off the main POM, they don't get
cleaned up; you'll need to `rm` those directories.
+If you are using `fish`, then `rm -rf */target` should do it.)
### Checking licenses
@@ -85,16 +103,24 @@ For modules which publish JARs (`slider-
run the third party check for JARs. If you run this at the top level project
it will do it for the relevant sub projects.
+Do this after building everything
- # in slider toplevel dir
- mvn license:add-third-party -Dlicense.useMissingFile
-
-You also need to verify that the slider-assembly module is compliant
-
- # in slider-assembly
- mvn license:aggregate-add-third-party -Dlicense.useMissingFile
-
-
+```
+# in slider toplevel dir
+mvn license:add-third-party -Dlicense.useMissingFile
+```
+
+This will generate license files under each module in
+`target/generated-sources/license/THIRD-PARTY.txt`. Inspect it and make sure
+there are no GPL or LGPL dependencies which do not also have a license option
+which is permitted by the ASF.
+
+You also need to verify that the slider-assembly and slider-agent modules are
compliant
+
+```
+# in slider-assembly and slider-agent
+mvn license:aggregate-add-third-party -Dlicense.useMissingFile
+```
This generates the metadata for licenses. Dependencies without a license are
warned about.
To fix this, make sure that the (generated or updated) file
`src/license/THIRD-PARTY.properties`
@@ -108,20 +134,23 @@ Verify your GPG key is provided to Apach
Apache verifies that distributions are correctly signed.
Login to [https://id.apache.org](https://id.apache.org) and verify the
fingerprint of GPG key used to sign above is provided. (`gpg --fingerprint`)
-Upload your GPG public key only to `/home/$ASF_USER/.pgpkey`
- scp public-key.asc people.apache.org:.pgpkey
+If you have no GPG key, add your fingerprint to the relevant field.
-*Tip*: if you are creating a new key, remember what password/passphrase you
have
+If you are creating a new key, remember what password/passphrase you have
given it. Then generate a revocation certificate and store it somewhere, so
that you can revoke the published key.
-##### Note: Profiles
+A new key will not be trusted by anyone else. You should introduce to the
+Apache web of trust by getting other developers to sign it. This can be done
+in person, or perhaps over a video conference in which key details can be
confirmed.
+
+##### Note: Selecting Hadoop versions
The slider POMs offer different profiles of Hadoop versions to build against.
The instructions
-below do not choose one. If a profile is needed, append it to all the maven
commands.
+below use the default. If a profile is needed, append it to all the maven
commands.
-e.g. `mvn clean install -Prelease-2.6`
+e.g. `mvn clean install -Prelease-2.7`
## Prepare
@@ -136,38 +165,45 @@ you can check this with `git remote -v`
* Verify that there's nothing not yet checked in
- git status
+```
+git status
+```
Everything must be checked in at this point.
* Sync up with the ASF repo. You do not want to release the wrong code.
- git checkout develop
- git pull origin
- git push origin
-
+```
+git checkout develop
+git pull origin
+git push origin
+```
### Start the JIRA
Create a JIRA for the release, estimate 12h
- export SLIDER_RELEASE_JIRA=SLIDER-544
+```
+export SLIDER_RELEASE_JIRA=SLIDER-544
+```
This seems pessimistic, but there may be surprises.
### Choose your versions
-Choose the version number for the release
+Choose the version number for the release and its successor for the
development branch
Example:
- 0.61.0-incubating
- 0.62.0-incubating-SNAPSHOT
+```
+0.61.0-incubating
+0.62.0-incubating-SNAPSHOT
- export SLIDER_RELEASE=0.61.0-incubating
- export SLIDER_DEV=0.62.0-incubating-SNAPSHOT
+export SLIDER_RELEASE=0.61.0-incubating
+export SLIDER_DEV=0.62.0-incubating-SNAPSHOT
+```
### Test that everything works
@@ -175,18 +211,24 @@ Do a final test run to make sure nothing
In the `slider` directory, run:
- mvn clean install -DskipTests
+```
+mvn clean install -DskipTests
+```
(Make sure the build isn't trying to pull down its JARs from the Apache Staging
repo)
Once everything is built- including `.tar` files, run the tests
- mvn test
+```
+mvn test
+```
Then the integration tests
- mvn integration-test -Dslider.conf.dir=${your-config-dir}
+```
+mvn integration-test -Dslider.conf.dir=${your-config-dir}
+```
This will run the functional tests as well as the `slider-core` tests.
@@ -205,8 +247,10 @@ Create or check out a release branch,
Example of creation then push to the origin
- git checkout -b branches/branch-0.6
- git push origin branches/branch-0.6
+```
+git checkout -b branches/branch-0.6
+git push origin branches/branch-0.6
+```
This branch is where all the release work will take place. That can include
patching and cherry picking from the development branch.
@@ -217,11 +261,15 @@ This branch is where all the release wor
Get back to the develop branch and patch its versions
- git checkout develop
- mvn versions:set -DnewVersion=$SLIDER_DEV
- git commit -a -m "$SLIDER_RELEASE_JIRA updating development POMs to
$SLIDER_DEV"
- git push origin
+```
+git checkout develop
+mvn versions:set -DnewVersion=$SLIDER_DEV
+git commit -a -m "$SLIDER_RELEASE_JIRA updating development POMs to
$SLIDER_DEV"
+
+git push origin
+```
+You may want to hold back from pushing things up if you are doing a dry run
# Automated releases via maven
@@ -292,8 +340,10 @@ In `~/.m2/settings.xml`:
It's handy to set up an environment with the arguments needed for releases.
- export RELARGS="-DskipTests -DskipITs -Papache-release"
- echo $RELARGS
+```
+export RELARGS="-DskipTests -DskipITs -Papache-release"
+echo $RELARGS
+```
The profile `apache-release` is used to select a subset of the app packages,
because the default builds of these weren't usable and had odd version numbers
@@ -313,66 +363,103 @@ Don't release from your development dir;
In a directory of your choice
- git clone https://git-wip-us.apache.org/repos/asf/incubator-slider.git
slider-$SLIDER_RELEASE
- cd slider-$SLIDER_RELEASE
-
+```
+git clone https://git-wip-us.apache.org/repos/asf/incubator-slider.git
slider-$SLIDER_RELEASE
+cd slider-$SLIDER_RELEASE
+```
+
### Check out the branch you plan to release on; tag it
The tag is so that if things go wrong you know what you have to roll back to.
- cd incubator-slider
- git checkout -t origin/branches/branch-0.6
+```
+git checkout -t origin/branches/branch-0.6
+```
In the new clone, tag the head of the branch you plan to to work from. You
don't need to push the tag to the repo.
- git tag -a tag_2015-01-27-prerelease-0.61.0 -m "prior to maven release
process"
-
-
+```
+git tag -a tag_2015-12-18-$SLIDER_RELEASE -m "starting release of
$SLIDER_RELEASE"
+```
-## Test Deploy a snapshot
+### Clean build it
-This pushes up a `-SNAPSHOT` version to the ASF snapshot repo.
-It doesn't change the POMs; it does verify that you have the basic upload
settings, signing keys etc.
+Maybe: purge your entire repository of slider artifacts
- mvn clean deploy $RELARGS -Darguments="$RELARGS"
+```
+rm -rf ~/.m2/repository/org/apache/slider
+```
+
+## Maven Release plugin
+
+The maven release plugin tries to do everything: update versions,
+tag repositories, push up artifacts.
+
+### Test Deploy a snapshot
+
+Create the -SNAPSHOT artifacts
+
+```
+mvn clean install -DskipTests
+```
-Assuming that all the test and integration tests have already been successful
on normal workspace, you can skip the tests
-
+
+Then push up a `-SNAPSHOT` version to the ASF snapshot repo.
+This dry-run verifies that you have the basic upload settings, signing keys
etc.
+
+```
+mvn clean deploy $RELARGS -Darguments="$RELARGS"
+```
After the upload has completed, the complete archives will be in the
version-specific subdirectory of
-
https://repository.apache.org/content/repositories/snapshots/org/apache/slider/slider-assembly/
+```
+https://repository.apache.org/content/repositories/snapshots/org/apache/slider/slider-assembly/
+```
+(there may be older versions there from previous dry runs: look for the one
with the date of your run on it)
Download the uploaded `.tar.gz`, expand it and verify that it is complete.
That includes
+1. Download the associate `.asc` file with the PGP signature.
+1. Validate the signature: `gpg --verify slider-assembly-*.asc`
+1. untar the assembly file: `tar -xz slider-assembly*.tar.gz`
1. Verify that the file `slider-agent.tar.gz` exists.
1. Execute `bin/slider version`
1. Assuming you have an existing installation for testing, set
`SLIDER_CONF_DIR` to the configuration directory, then
-`slider start <INSTANCE>` to start the relevant cluster instance.
-
-
-## Perform a dry run of the release
+`./slider diagnostics --yarn --client --credentials --verbose`
+
-Perform a dry run. This will commit changes to your local branch.
+### Perform a dry run of the release
- mvn release:prepare -DdryRun=true -DautoVersionSubmodules=true
-Darguments="$RELARGS"
+Perform a dry run. This will prompt you for new versions âand commit changes
to your local branch.
+```
+mvn release:prepare -DdryRun=true -DautoVersionSubmodules=true
-Darguments="$RELARGS"
+```
1. This prompts for the version numbers for the slider version to release: use
the `slider-0.61.0-incubating` or equivalent
-1. Use the same (as recommanded)
-go with them unless you have specific choices of
+1. go with them unless you have specific choices of
versions.
1. Look at the output to verify the versions are what you want.
-1. Roll back to the commit tagged earlier
+1. Roll back to the commit tagged earlier: `git reset --hard`
## Perform the Maven release
- mvn release:clean
- mvn release:prepare -DautoVersionSubmodules=true -Darguments="$RELARGS"
-Dresume=false
- mvn release:perform -Darguments="$RELARGS"
+```
+mvn release:clean
+
+mvn -B release:prepare -DautoVersionSubmodules=true
-Dtag=slider-$SLIDER_RELEASE \
+ -DreleaseVersion=$SLIDER_RELEASE \
+ -DdevelopmentVersion=$SLIDER_DEV \
+ -Darguments="$RELARGS" -Dresume=false
+
+mvn release:perform -Darguments="$RELARGS"
+
+
+```
1. The `release:prepare` command will trigger the local commit of changes to
POMs, tagging this and pushing up the branch.
2. Only skip those test runs if you have been running the tests in advance.
@@ -393,7 +480,9 @@ After this phase (or when you cancel it)
Try
- mvn release:rollback
+```
+mvn release:rollback
+```
It *may* work. Then
@@ -404,10 +493,13 @@ changing things. That should be the revi
1. Run `mvn clean`
-If you are in a specific release-only directory (as per the instructions):
+If you are in a specific release-only directory (as recommended):
+
+```
+git clean -d -f -x
+git reset --hard
+```
- git clean -d -f -x
-
### Tip: branching instead of preparing
The `mvn release:branch` goal can be used instead of `mvn release:prepare`,
@@ -421,24 +513,40 @@ There's a small projects [slider-depende
which verifies that slider artifacts can be downloaded from the public or
staging repositories.
Clone this project and read its instructions to verify that everything went up
to the staging repo.
+### Tip: isolating the release process from the ASF repo
+
+While trying to get the release out, the maven release plugin likes to commit
things
+to git, create tags and push them out. You can isolate your work from the ASF
repository
+by (temporarily) unhooking your release source tree from the ASF one.
+You also need to isolate the repository settings in the build
+
+Add these options to the build commands, using a URL to a repository to
+which you do have access
+
+```
+-Pprivate-git-repo
-Dgit.repo=https://github.com/steveloughran/incubator-slider.git
+```
+
+After creating a release to put up for a vote, you will need to push the
+relevant (final) tag out to the ASF repository.
## Close the release in Nexus
-1. log in to
[https://repository.apache.org/index.html](https://repository.apache.org/index.html)
+1. Log in to
[https://repository.apache.org/index.html](https://repository.apache.org/index.html)
with your ASF username and LDAP password
-1. go to [Staging
Repositories](https://repository.apache.org/index.html#stagingRepositories)
-1. find the latest slider repository in the list
-1. select it;
+1. Go to [Staging
Repositories](https://repository.apache.org/index.html#stagingRepositories)
+1. Find the latest slider repository in the list
+1. Select it;
1. Browse the content, verify the version number is what you want.
-1. hit the "close" button. This will fail if it doesn't meet the ASF criteria
+1. Hit the "close" button. This will fail if it doesn't meet the ASF criteria
1. Wait for the close process to complete.
1. Get the URL of the closed staging repository. This is needed for the next
step âthe votes.
## Tell JIRA of the releases
-
+
Tell JIRA there's a new release. This moves it from "unreleased" to
"released", creating a URL listing all issues fixed.
This URL can be used in the vote emails to highlight changes.
@@ -447,15 +555,15 @@ This URL can be used in the vote emails
Slider project on JIRA.
1. Go to the [Slider Versions
page](https://issues.apache.org/jira/plugins/servlet/project-config/SLIDER/versions)
-2. Locate the pending release. Make sure it is in the correct order of
releases: all previous releases MUST be below it; future releases above it. The
immediate next release MUST be directly above the pending release.
+1. Locate the pending release. Make sure it is in the correct order of
releases: all previous releases MUST be below it; future releases above it. The
immediate next release MUST be directly above the pending release.
1. On the pending release
i. Set the release date to the current day.
i. Select the drop down "gear-cog" menu to the right of the page, then
the "Release" option.
i. Carrying all forward issues that haven't been completed *except the
JIRA covering the release itself.
1. Go to the [Slider versions
page](https://issues.apache.org/jira/browse/SLIDER/?selectedTab=com.atlassian.jira.jira-projects-plugin:versions-panel)
-2. Locate the release just issued and click through it.
+1. Locate the release just issued and click through it.
1. Get the page link
-2. Ideally, shorten it via the [ASF Link shortener](https://s.apache.org/)
+1. Ideally, shorten it via the [ASF Link shortener](https://s.apache.org/)
If the vote is cancelled, the release can be "unreleased".
@@ -466,26 +574,33 @@ If the vote is cancelled, the release ca
set up the environment with the git release tag, URL to the staging
repository, and your ASF username:
- export RELEASE_TAG=
- export STAGING_REPO=
- export ASF_USER=
- export SOURCE_ZIP
- export SOURCE_TAR_GZ
+```
+export RELEASE_TAG=
+export STAGING_REPO=
+export ASF_USER=
+export SOURCE_ZIP
+export SOURCE_TAR_GZ
+```
example
- export RELEASE_TAG=64a8bac068e6801748fb973dbfb590bc62c60935
-
- export
STAGING_REPO=https://repository.apache.org/content/repositories/orgapacheslider-1003/
+```
+export RELEASE_TAG=64a8bac068e6801748fb973dbfb590bc62c60935
+export ASF_USER=stevel
- export ASF_USER=stevel
+export
STAGING_REPO=https://repository.apache.org/content/repositories/orgapacheslider-1003
+curl --head $STAGING_REPO
- export
SOURCE_ZIP=https://repository.apache.org/content/repositories/orgapacheslider-1003/org/apache/slider/slider/$SLIDER_RELEASE/slider-$SLIDER_RELEASE-source-release.tar.gz
+export
SOURCE_ZIP=$STAGING_REPO/org/apache/slider/slider/$SLIDER_RELEASE/slider-$SLIDER_RELEASE-source-release.tar.gz
+echo $SOURCE_ZIP
+curl --head $SOURCE_ZIP
- export
SOURCE_TAR_GZ=https://repository.apache.org/content/repositories/orgapacheslider-1003/org/apache/slider/slider/$SLIDER_RELEASE/slider-$SLIDER_RELEASE-source-release.zip
+export
SOURCE_TAR_GZ=$STAGING_REPO/org/apache/slider/slider/$SLIDER_RELEASE/slider-$SLIDER_RELEASE-source-release.zip
+echo $SOURCE_TAR_GZ
- export APACHE_VERSION_PAGE=(shorten this via
s.apache.org)https://issues.apache.org/jira/browse/SLIDER/fixforversion/12327198
+export APACHE_VERSION_PAGE=(shorten this via
s.apache.org)https://issues.apache.org/jira/browse/SLIDER/fixforversion/12327198
+```
### Call for a release vote on the dev@slider list
@@ -664,7 +779,7 @@ the zip and tar files and their checksum
svn co https://dist.apache.org/repos/dist/release/incubator/slider
slider-dist
cd slider-dist
-
+
Make sure your PGP keys is in the `KEYS` file. The instructions to add a key
are at the top of that file
@@ -758,4 +873,11 @@ Check out the develop branch and purge a
mvn clean
-
\ No newline at end of file
+
+## Manual "we don't trust `mvn release` strategy
+
+
+```
+mvn versions:set -DnewVersion=$SLIDER_RELEASE
+
+```
