I'll start taking a look at the maven issue. We might not want to use maven release plugin given that we control the version number already across this repository via other means.
On Wed, Jul 31, 2019 at 4:26 PM Sutou Kouhei <k...@clear-code.com> wrote: > Hi, > > Sorry for not replying this thread. > > I think that the biggest problem is related to our Java > package. > > > We'll be able to resolve the GPG key problem by creating a > GPG key only for nightly release test. We can share the test > GPG key publicly because it's a just for testing. > > It'll work for our binary artifacts and APT/Yum repositories > but not work for our Java package. I don't know where GPG > key is used in our Java package... > > > We'll be able to resolve the Git commit problem by creating > a cloned Git repository for test. It's done in our > dev/release/00-prepare-test.rb[1]. > > [1] > https://github.com/apache/arrow/blob/master/dev/release/00-prepare-test.rb#L30 > > The biggest problem for the Git commit is our Java package > requires "apache-arrow-${VERSION}" tag on > https://github.com/apache/arrow . (Right?) > I think that "mvm release:perform" in > dev/release/01-perform.sh does so but I don't know the > details of "mvm release:perform"... > > > More details: > > dev/release/00-prepare.sh: > > We'll be able to run this automatically when we can resolve > the above GPG key problem in our Java package. We can > resolve the Git commit problem by creating a cloned Git > repository. > > dev/release/01-prepare.sh: > > We'll be able to run this automatically when we can resolve > the above Git commit ("apche-arrow-${VERSION}" tag) problem > in our Java package. > > dev/release/02-source.sh: > > We'll be able to run this automatically by creating a GPG > key for nightly release test. We'll use Bintray to upload RC > source archive instead of dist.apache.org. Ah, we need a > Bintray API key for this. It must be secret. > > dev/release/03-binary.sh: > > We'll be able to run this automatically by creating a GPG > key for nightly release test. We need a Bintray API key too. > > We need to improve this to support nightly release test. It > use "XXX-rc" such as "debian-rc" for Bintray "package" name. > It should use "XXX-nightly" such as "debian-nightly" for > nightly release test instead. > > dev/release/post-00-release.sh: > > We'll be able to skip this. > > dev/release/post-01-upload.sh: > > We'll be able to skip this. > > dev/release/post-02-binary.sh: > > We'll be able to run this automatically by creating Bintray > "packages" for nightly release and use them. We can create > "XXX-nightly-release" ("debian-nightly-release") Bintray > "packages" and use them instead of "XXX" ("debian") Bintray > "packages". > > "debian" Bintray "package": https://bintray.com/apache/debian/ > > We need to improve this to support nightly release. > > dev/release/post-03-website.sh: > > We'll be able to run this automatically by creating a cloned > Git repository for test. > > It's better that we have a Web site to show generated pages. > We can create > https://github.com/apache/arrow-site/tree/asf-site/nightly > and use it but I don't like it. Because arrow-site increases > a commit day by day. > Can we prepare a Web site for this? (arrow-nightly.ursalabs.org?) > > dev/release/post-04-rubygems.sh: > > We may be able to use GitHub Package Registry[2] to upload > RubyGems. We can use "pre-release" package feature of > https://rubygems.org/ but it's not suitable for > nightly. It's for RC or beta release. > > [2] https://github.blog/2019-05-10-introducing-github-package-registry/ > > dev/release/post-05-js.sh: > > We may be able to use GitHub Package Registry[2] to upload > npm packages. > > dev/release/post-06-csharp.sh: > > We may be able to use GitHub Package Registry[2] to upload > NuGet packages. > > dev/release/post-07-rust.sh: > > I don't have any idea. But it must be ran > automatically. It's always failed. I needed to run each > command manually. > > dev/release/post-08-remove-rc.sh: > > We'll be able to skip this. > > > Thanks, > -- > kou > > In <CAJPUwMAmp0jvz6qrdqehBEUB_NdbGEsHFNgLW9Q6V9RFTnk=3...@mail.gmail.com> > "Re: [DISCUSS] Release cadence and release vote conventions" on Wed, 31 > Jul 2019 15:35:57 -0500, > Wes McKinney <wesmck...@gmail.com> wrote: > > > The PMC member and their GPG keys need to be in the loop at some > > point. The release artifacts can be produced by some kind of CI/CD > > system so long as the PMC member has confidence in the security of > > those artifacts before signing them. For example, we build the > > official binary packages on public CI services and then download and > > sign them with Crossbow. I think the same could be done in theory with > > the source release but we'd first need to figure out what to do about > > the parts that create git commits. > > > > On Wed, Jul 31, 2019 at 11:23 AM Andy Grove <andygrov...@gmail.com> > wrote: > >> > >> To what extent would it be possible to automate the release process via > >> CICD? > >> > >> On Wed, Jul 31, 2019 at 9:19 AM Wes McKinney <wesmck...@gmail.com> > wrote: > >> > >> > I think one thing that would help would be improving the > >> > reproducibility of the source release process. The RM has to have > >> > their machine configured in a particular way for it to work. > >> > > >> > Before anyone says "Docker" it isn't an easy solution because the > >> > release scripts need to be able to create git commits (created by the > >> > Maven release plugin) and sign artifacts using the RM's GPG keys. > >> > > >> > On Sat, Jul 27, 2019 at 10:04 PM Micah Kornfield < > emkornfi...@gmail.com> > >> > wrote: > >> > > > >> > > I just wanted to bump this thread. Kou and Krisztián as the last > two > >> > > release managers is there any specific infrastructure that you think > >> > might > >> > > have helped? > >> > > > >> > > Thanks, > >> > > Micah > >> > > > >> > > On Wed, Jul 17, 2019 at 11:29 PM Micah Kornfield < > emkornfi...@gmail.com> > >> > > wrote: > >> > > > >> > > > I'd can help as well, but not exactly sure where to start. It > seems > >> > like > >> > > > there are already some JIRAs opened [1] > >> > > > for improving the release? Could someone more familiar with the > >> > process > >> > > > pick out the highest priority ones? Do more need to be opened? > >> > > > > >> > > > Thanks, > >> > > > Micah > >> > > > > >> > > > [1] > >> > > > > >> > > https://issues.apache.org/jira/browse/ARROW-2880?jql=project%20%3D%20ARROW%20AND%20status%20in%20(Open%2C%20%22In%20Progress%22%2C%20Reopened)%20AND%20component%20in%20(%22Developer%20Tools%22%2C%20Packaging)%20and%20summary%20~%20Release > >> > > > > >> > > > On Sat, Jul 13, 2019 at 7:17 AM Wes McKinney <wesmck...@gmail.com > > > >> > wrote: > >> > > > > >> > > >> To be effective at improving the life of release managers, the > nightly > >> > > >> release process really should use as close as possible to the > same > >> > > >> scripts that the RM uses to produce the release. Otherwise we > could > >> > > >> have a situation where the nightlies succeed but there is some > problem > >> > > >> that either fails an RC or is unable to be produced at all. > >> > > >> > >> > > >> On Sat, Jul 13, 2019 at 9:12 AM Andy Grove < > andygrov...@gmail.com> > >> > wrote: > >> > > >> > > >> > > >> > I would like to volunteer to help with Java and Rust release > process > >> > > >> work, > >> > > >> > especially nightly releases. > >> > > >> > > >> > > >> > Although I'm not that familiar with the Java implementation of > >> > Arrow, I > >> > > >> > have been using Java and Maven for a very long time. > >> > > >> > > >> > > >> > Do we envisage a single nightly release process that releases > all > >> > > >> languages > >> > > >> > simultaneously? or do we want separate process per language, > with > >> > > >> different > >> > > >> > maintainers? > >> > > >> > > >> > > >> > > >> > > >> > > >> > > >> > On Wed, Jul 10, 2019 at 8:18 AM Wes McKinney < > wesmck...@gmail.com> > >> > > >> wrote: > >> > > >> > > >> > > >> > > On Sun, Jul 7, 2019 at 7:40 PM Sutou Kouhei < > k...@clear-code.com> > >> > > >> wrote: > >> > > >> > > > > >> > > >> > > > Hi, > >> > > >> > > > > >> > > >> > > > > in future releases we should > >> > > >> > > > > institute a minimum 24-hour "quiet period" after any > community > >> > > >> > > > > feedback on a release candidate to allow issues to be > examined > >> > > >> > > > > further. > >> > > >> > > > > >> > > >> > > > I agree with this. I'll do so when I do a release manager > in > >> > > >> > > > the future. > >> > > >> > > > > >> > > >> > > > > To be able to release more often, two things have to > happen: > >> > > >> > > > > > >> > > >> > > > > * More PMC members must engage with the release > management > >> > role, > >> > > >> > > > > process, and tools > >> > > >> > > > > * Continued improvements to release tooling to make the > >> > process > >> > > >> less > >> > > >> > > > > painful for the release manager. For example, it seems > we may > >> > > >> want to > >> > > >> > > > > find a different place than Bintray to host binary > artifacts > >> > > >> > > > > temporarily during release votes > >> > > >> > > > > >> > > >> > > > My opinion that we need to build nightly release system. > >> > > >> > > > > >> > > >> > > > It uses dev/release/NN-*.sh to build .tar.gz and binary > >> > > >> > > > artifacts from the .tar.gz. > >> > > >> > > > It also uses dev/release/verify-release-candidate.* to > >> > > >> > > > verify build .tar.gz and binary artifacts. > >> > > >> > > > It also uses dev/release/post-NN-*.sh to do post release > >> > > >> > > > tasks. (Some tasks such as uploading a package to packaging > >> > > >> > > > system will be dry-run.) > >> > > >> > > > > >> > > >> > > > >> > > >> > > I agree that having a turn-key release system that's capable > of > >> > > >> > > producing nightly packages is the way to do. That way any > problems > >> > > >> > > that would block a release will come up as they happen > rather than > >> > > >> > > piling up until the very end like they are now. > >> > > >> > > > >> > > >> > > > I needed 10 or more changes for dev/release/ to create > >> > > >> > > > 0.14.0 RC0. (Some of them are still in my local stashes. I > >> > > >> > > > don't have time to create pull requests for them > >> > > >> > > > yet. Because I postponed some tasks of my main > >> > > >> > > > business. I'll create pull requests after I finished the > >> > > >> > > > postponed tasks of my main business.) > >> > > >> > > > > >> > > >> > > > >> > > >> > > Thanks. I'll follow up on the 0.14.1/0.15.0 thread -- since > we > >> > need to > >> > > >> > > release again soon because of problems with 0.14.0 please > let us > >> > know > >> > > >> > > what patches will be needed to make another release. > >> > > >> > > > >> > > >> > > > If we fix problems related to dev/release/ in our normal > >> > > >> > > > development process, release process will be less painful. > >> > > >> > > > > >> > > >> > > > The biggest problem for 0.14.0 RC0 is java/pom.xml related: > >> > > >> > > > https://github.com/apache/arrow/pull/4717 > >> > > >> > > > > >> > > >> > > > It was difficult for me because I don't have Java > >> > > >> > > > knowledge. Release manager needs help from many developers > >> > > >> > > > because release manager may not have knowledge of all > >> > > >> > > > supported languages. Apache Arrow supports 10 over > >> > > >> > > > languages. > >> > > >> > > > > >> > > >> > > > > >> > > >> > > > For Bintray API limit problem, we'll be able to resolve it. > >> > > >> > > > I was added to https://bintray.com/apache/ members: > >> > > >> > > > > >> > > >> > > > https://issues.apache.org/jira/browse/INFRA-18698 > >> > > >> > > > > >> > > >> > > > I'll be able to use Bintray API without limitation in the > >> > > >> > > > future. Release managers should also request the same > thing. > >> > > >> > > > > >> > > >> > > > >> > > >> > > This is good, I will add myself. Other PMC members should > also add > >> > > >> > > themselves. > >> > > >> > > > >> > > >> > > > > >> > > >> > > > Thanks, > >> > > >> > > > -- > >> > > >> > > > kou > >> > > >> > > > > >> > > >> > > > In <CAJPUwMBRzYQ=hbVwFuPYAB-O= > >> > > >> lsowxqxidjapc_cofguksj...@mail.gmail.com> > >> > > >> > > > "[DISCUSS] Release cadence and release vote conventions" > on > >> > Sat, > >> > > >> 6 Jul > >> > > >> > > 2019 16:28:50 -0500, > >> > > >> > > > Wes McKinney <wesmck...@gmail.com> wrote: > >> > > >> > > > > >> > > >> > > > > hi folks, > >> > > >> > > > > > >> > > >> > > > > As a reminder, particularly since we have many new > community > >> > > >> members > >> > > >> > > > > (some of whom have never been involved with an ASF > project > >> > > >> before), > >> > > >> > > > > releases are approved exclusively by the PMC and in > general > >> > > >> releases > >> > > >> > > > > cannot be vetoed. In spite of that, we strive to make > releases > >> > > >> that > >> > > >> > > > > have unanimous (either by explicit +1 or lazy consent) > >> > support of > >> > > >> the > >> > > >> > > > > PMC. So it is better to have unanimous 5 +1 votes than 6 > +1 > >> > votes > >> > > >> with > >> > > >> > > > > a -1 dissenting vote. > >> > > >> > > > > > >> > > >> > > > > On the 0.14.0 vote, as with previous release votes, some > >> > issues > >> > > >> with > >> > > >> > > > > the release were raised by members of the community, > whether > >> > > >> build or > >> > > >> > > > > test-related problems or other failures. Technically > speaking, > >> > > >> such > >> > > >> > > > > issues have no _direct_ bearing on whether a release vote > >> > passes, > >> > > >> only > >> > > >> > > > > on whether PMC members vote +1, 0, or -1. A PMC member is > >> > allowed > >> > > >> to > >> > > >> > > > > change their vote based on new information -- for > example, if > >> > I > >> > > >> voted > >> > > >> > > > > +1 on a release and then someone reported a serious > licensing > >> > > >> issue, > >> > > >> > > > > then I would revise my vote to -1. > >> > > >> > > > > > >> > > >> > > > > On the RC0 vote thread, Jacques wrote [1] > >> > > >> > > > > > >> > > >> > > > > "A release vote should last until we arrive at consensus. > >> > When an > >> > > >> > > > > issue is potentially identified, those that have voted > should > >> > be > >> > > >> given > >> > > >> > > > > ample time to change their vote and others that may have > been > >> > lazy > >> > > >> > > > > consenters should be given time to chime in. There is no > >> > maximum > >> > > >> > > > > amount of time a vote can be open. Allowing at least 24 > hours > >> > > >> after an > >> > > >> > > > > objection is raised is a pretty minimum expectation > unless the > >> > > >> > > > > objector removes their objection. > >> > > >> > > > > > >> > > >> > > > > Note that Apache is more focused on consensus than > timing (as > >> > > >> opposed > >> > > >> > > to > >> > > >> > > > > virtually other other organizations in the world)." > >> > > >> > > > > > >> > > >> > > > > I agree with this and my opinion is that in future > releases we > >> > > >> should > >> > > >> > > > > institute a minimum 24-hour "quiet period" after any > community > >> > > >> > > > > feedback on a release candidate to allow issues to be > examined > >> > > >> > > > > further. If someone finds a potential problem, and no > negative > >> > > >> votes > >> > > >> > > > > are cast or changed, then the vote can close. > >> > > >> > > > > > >> > > >> > > > > As a related matter, it seems clear to me that Apache > Arrow > >> > should > >> > > >> > > > > have more frequent releases. I think this would decrease > >> > pressure > >> > > >> on > >> > > >> > > > > developers and users alike. While we've made strides to > >> > improve > >> > > >> the > >> > > >> > > > > tooling for release management (big thanks to Kou, > Yosuke, > >> > > >> Krisztian, > >> > > >> > > > > and others), there is still quite some labor involved and > >> > > >> potential > >> > > >> > > > > for issues (e.g. API rate limiting for binary artifacts > on > >> > > >> Bintray). > >> > > >> > > > > > >> > > >> > > > > To be able to release more often, two things have to > happen: > >> > > >> > > > > > >> > > >> > > > > * More PMC members must engage with the release > management > >> > role, > >> > > >> > > > > process, and tools > >> > > >> > > > > * Continued improvements to release tooling to make the > >> > process > >> > > >> less > >> > > >> > > > > painful for the release manager. For example, it seems > we may > >> > > >> want to > >> > > >> > > > > find a different place than Bintray to host binary > artifacts > >> > > >> > > > > temporarily during release votes > >> > > >> > > > > > >> > > >> > > > > Any other ideas for things we can do to improve the > process > >> > and > >> > > >> > > > > cadence of releases? > >> > > >> > > > > > >> > > >> > > > > Thanks, > >> > > >> > > > > Wes > >> > > >> > > > > > >> > > >> > > > > [1]: > >> > > >> > > > >> > > >> > >> > > https://lists.apache.org/thread.html/be6210e97b838494a5516dad6408f479efe4c98aff805000597c0196@%3Cdev.arrow.apache.org%3E > >> > > >> > > > >> > > >> > >> > > > > >> > >
