I agree with Jihoon that deciding what should and shouldn't be in a release on a feature by feature basis is way too expensive to do at release time. I think the PR that adds a thing is the most appropriate venue to discuss whether or not a change should be accepted to a release, unless a major issue is discovered after being merged. Even then reverting instead of fixing in place should be a relatively exceptional circumstance when there is a major issue and a fix would take a long time for a release that is otherwise ready. It is healthy to review changes while voting on a release as a sort of spot check (I tend to do it whenever I am release manager while searching the set of changes to determine if anything should be in the release notes which was not tagged), since it can help to uncover any of these potential issues that might have been missed, but anything nearing formally voting on every feature sounds tedious.
With regards to integration tests, I don't think it is very realistic to expect voters to be running the integration tests across the entire set of platforms, access to cloud infrastructure or a hadoop cluster should not be a barrier to approving a release or not. Not to mention it takes hours and hours of compute time to run the complete suite. I don't really have a good solution to this. On Mon, May 3, 2021 at 10:38 PM Jihoon Son <jihoon...@apache.org> wrote: > Thanks Suneet. All look good to me except the ones below. > > > - Verify release notes contains a note about all issues marked with > release notes label > > I personally treat the "release notes" label as a mark for the > candidate PRs that should be noted in the release notes. I sometimes > intentionally drop some notes if the release notes grow too large when > I'm the release manager. I'm not sure what others do when they are the > release manager but think this is OK. > > > - [Druid requirement] Vote on whether any features are being released > that are incomplete - missing integration tests, un-expected change in > behavior, etc. > > Can you elaborate more on this suggestion? Are you suggesting making > it a part of the release vote process? Or is it voting for each > feature before starting the release vote? > To me, making it a part of the release vote process seems too > expensive because the release vote can start only when the tag is > ready. I would like to have such discussions before the release vote > is started. > > Today, we have been having such discussions in each PR when someone > thinks a particular PR should be or should not be in the release. > These discussions usually do not involve a vote because we usually > reach a consensus after discussion. > > > Does a person voting need to run > these integration tests today, since they are not part of the automated > pipeline? > > This is a good point. I think we should run them manually for now. In > the long term, we should find a way to automate those integration > tests. > > On Wed, Apr 28, 2021 at 10:18 PM Suneet Saldanha > <suneet.salda...@imply.io> wrote: > > > > Hi Jihoon, > > > > Thanks for pointing that out. Then, I'd update my proposal to include > this > > step as something that always needs to be done manually. > > > > > What I think should always be done manually: > > > > > > - [ASF requirement] download all signed source code packages, > validate > > checksums + signatures, compile without tests. start a Druid cluster and > > ingest wikipedia data onto a micro-quickstart cluster and run a > > simple query against it. > > > - [Druid requirement] Vote on whether any features are being > released > > that are incomplete - missing integration tests, un-expected change in > > behavior, etc. > > > - [Druid requirement] Verify that all the automated tests that run > for > > the tag have actually passed (maybe this can be automated too?). I > imagine > > this step is to look at travis for the tag to ensure that all the tests > > actually passed. > https://travis-ci.com/github/apache/druid/builds/224182054 > > - this is the travis build for the 0.21.0 tag (which appears to be > failing) > > > > If we get agreement on this, I'll be happy to add / update a doc that > talks > > about what specifically is expected from someone voting on an Apache > Druid > > release. > > > > While writing out this proposal below, I've realized we have a set of > > integration tests that do not run as part of the CI pipeline because they > > require credentials to some third party systems (eg. Kinesis ITs) or we > > can't get the tests to consistently run in the Travis pipeline without > > intermittent failures (eg. Hadoop ITs). Does a person voting need to run > > these integration tests today, since they are not part of the automated > > pipeline? > > > > > > > > > > > > > > On Mon, Apr 26, 2021 at 12:30 PM Jihoon Son <jihoon...@apache.org> > wrote: > > > > > Hi Suneet, thanks for your suggestion. > > > > > > However, the Apache release policy [1] explicitly says the below. > > > > > > > Before casting +1 binding votes, individuals are REQUIRED to download > > > all signed source code packages onto their own hardware, verify that > they > > > meet all requirements of ASF policy on releases as described below, > > > validate all cryptographic signatures, compile as provided, and test > the > > > result on their own platform. > > > > > > Because only the source release is our official release, PMCs have to > > > download at least the source code package and verify it manually per > > > this requirement. > > > That being said, we can automate only verifying the binary package and > > > docker image at this moment. > > > > > > Jihoon > > > > > > [1] https://www.apache.org/legal/release-policy.html#policy > > > > > > On Fri, Apr 23, 2021 at 6:26 PM Suneet Saldanha > > > <suneet.salda...@imply.io> wrote: > > > > > > > > Hi Druids, > > > > > > > > I was thinking about the 0.21.0 release vote that was just > happening, and > > > > found it tedious to help with the vote because I needed to build the > code > > > > from source as well as run all the tests on my local machine. This > got me > > > > thinking about what are the key steps that we should check manually > in > > > the > > > > release process and what steps can we rely on automation that exists > in > > > > Travis today. I don't have full context on what steps are needed, > but I > > > > thought I'd start a thread to hear from everyone about what steps > should > > > be > > > > done before a release, and whether any of them need to be done > manually, > > > or > > > > if we could leverage our CI pipelines to replace these steps. > > > > > > > > What I think should be done before the release? > > > > > > > > - Run all unit tests > > > > - RAT License checks > > > > - Run all integration tests > > > > - Verify checksums > > > > - Verify signatures > > > > - Verify release notes contain > > > > > > > > What I think should be done manually (because automation doesn't > exist > > > > today): > > > > > > > > - Verify docker pull works > > > > - Verify Druid can start from the pulled docker image > > > > - Verify no open issues on github with the milestone for the > version > > > > being released > > > > - Verify release notes contains a note about all issues marked > with > > > > release notes label > > > > > > > > > > > > What I think should always be done manually: > > > > > > > > - Vote on whether any features are being released that are > incomplete > > > > - Verify that all the automated tests that run for the tag have > > > actually > > > > passed (maybe this can be automated too?) > > > > > > > > Look forward to hearing thoughts on this :) > > > > > > --------------------------------------------------------------------- > > > To unsubscribe, e-mail: dev-unsubscr...@druid.apache.org > > > For additional commands, e-mail: dev-h...@druid.apache.org > > > > > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: dev-unsubscr...@druid.apache.org > For additional commands, e-mail: dev-h...@druid.apache.org > >
