Thank you all for feedback. I filed BEAM-7242 for myself to update the release process with sufficient tooling to add support for this. I am unlikely to be able to do it for the current release but hopefully I will work on it soon.
*From: *Michael Luckey <adude3...@gmail.com> *Date: *Mon, May 6, 2019 at 4:48 PM *To: * <dev@beam.apache.org> Thanks Ahmet for the time you put into this. AFAIU Roberts proposal > resolves my concerns. > > On Mon, May 6, 2019 at 8:01 PM Ahmet Altay <al...@google.com> wrote: > >> Thank you Max. Michael, does the current state of the proposal address >> your concerns? >> >> *From: *Maximilian Michels <m...@apache.org> >> *Date: *Mon, May 6, 2019 at 8:32 AM >> *To: * <dev@beam.apache.org> >> >> Thanks for the link to the distribution guideline thread. The proposed >>> process sounds good to me. >>> >>> Concerning Michael's point, I think we should follow the same procedure >>> as we do for releasing to Maven Central, i.e. create a staging >>> repository for validation which then gets released. >>> >>> The easiest way to do that, is to directly use the artifacts that we >>> create during the release process: >>> >>> pip install https://dist.apache.org/repos/dist\ >>> /dev/beam/<version>/python >>> >>> We wouldn't have to built a second artifact or change a version number. >>> However, I understand if we want the additional convenience of: >>> >>> pip install --pre apache_beam >>> >>> As Robert mentioned, this would just be the additional step to rename >>> the version number and push the RC package to PyPi. Not necessary in my >>> opinion but I don't mind. >>> >> >> Value proposition is for downstream library providers. They can update >> their setup.py file to test beam rc in combination with rest of their other >> dependencies and test that combination with ease. I agree it is not >> necessary for us or any other party that just wants to test beam packages >> and its direct dependencies in isolation. >> >> >>> >>> Thanks, >>> Max >>> >>> On 02.05.19 19:43, Ahmet Altay wrote: >>> > >>> > >>> > On Thu, May 2, 2019 at 9:29 AM Robert Bradshaw <rober...@google.com >>> > <mailto:rober...@google.com>> wrote: >>> > >>> > On Thu, May 2, 2019 at 6:03 PM Michael Luckey <adude3...@gmail.com >>> > <mailto:adude3...@gmail.com>> wrote: >>> > > >>> > > Yes, I understood this. But I m personally more paranoid about >>> > releasing. >>> > > >>> > > So formally vote (and corresponding testing) was done on rc. If >>> > we rebuild and resign, wouldn't that mean we also need to revote? >>> > >>> > Yeah, that's the sticking point. I suppose we could build the >>> packages >>> > with rc tags, push them to pypi, and also build them without rc >>> tags, >>> > and push those (and the full source tarball, which doesn't have an >>> rc >>> > tag either) to svn, and have the vote officially cover what's in >>> svn >>> > but the rc ones are just for convenience. >>> > >>> > >>> > I prefer this way of double building. I think modifying the release >>> > process as in your initial proposal would be risky. In that proposal, >>> > what we test for validations would be different than what we push out >>> as >>> > final releases. The modification I see would be: >>> > >>> > 2c) Get the artifacts from svn, un-package, change the version to >>> > include the rc number, and re-package and push. (This is very similar >>> to >>> > the process we use for building the wheel files today, with the >>> > exception of changing the version part.) >>> > >>> > (But, given that I can "pip >>> > install https::svn.apache.org/path/to/tarball >>> > <http://svn.apache.org/path/to/tarball>" it'd primarily have >>> > value for others doing "pip install --pre".) >>> > >>> > >>> > This is regardless of whether is OK per apache to publish such >>> binary >>> > blobs to a third party place (though IMHO it follows the intent of >>> the >>> > release process). >>> > >>> > >>> > This is also my understanding. What is being proposed here aligns with >>> > what Kenn found out in the other threads. >>> > >>> > >>> > > If I understand correctly, there will be some changed version >>> > string in distributed sources (setup.py?). So there is some binary >>> > difference. And just talking about me, doing that repackaging I >>> > would certainly mess it up and package some unwanted changes. >>> > >>> > We definitely would not want this to be a manual step--I wouldn't >>> > trust myself :). >>> > >>> > >>> > > On Thu, May 2, 2019 at 5:43 PM Robert Bradshaw >>> > <rober...@google.com <mailto:rober...@google.com>> wrote: >>> > >> >>> > >> On Thu, May 2, 2019 at 5:24 PM Michael Luckey >>> > <adude3...@gmail.com <mailto:adude3...@gmail.com>> wrote: >>> > >> > >>> > >> > Thanks Ahmet for calling out to the airflow folks. I believe, >>> > I am able to follow their argument. So from my point of view I do >>> > not have an issue with apache policy. But honestly still trying to >>> > wrap my head around Roberts concern with rebuilding/resigning. >>> > Currently, our actual release is only a tag on source repo and >>> > promoting artefacts. Do not yet understand how that needs to change >>> > to get PyPi included. >>> > >> >>> > >> It's not a big change, but let me clarify. >>> > >> >>> > >> Currently our release preparation goes something like this: >>> > >> >>> > >> 1) Check out the repo, update the versions to 2.x, build and >>> > sign the artifacts. >>> > >> 2) Announce these artifacts as rcN >>> > >> 2a) Push the artifacts to SVN dev/... >>> > >> 2b) Push artifacts to the apache maven repository. >>> > >> 3) Depending on vote, go back to step (1) or forward to step >>> (4). >>> > >> 4) Copy these artifacts as the actual release. >>> > >> >>> > >> Now if we just try to add (2c) Push these artifacts to Pypi, >>> it will >>> > >> be treated (by pypi's tooling, anyone who downloads the >>> tarball, >>> > ...) >>> > >> as an actual release. You also can't re-push a tarball with >>> the same >>> > >> name and different contents (the idea being that named releases >>> > should >>> > >> never change). So we'd need to change step (1) to update the >>> version >>> > >> to 2.x.rcN *and* add a step in (4) to update the version to 2.x >>> > (no rc >>> > >> suffix), rebuild, resign before publishing. >>> > >> >>> > >> As mentioned, possibly the rcN suffix could be part of the >>> building >>> > >> step for Python. >>> > >> >>> > >> > On Wed, May 1, 2019 at 1:33 AM Ahmet Altay <al...@google.com >>> > <mailto:al...@google.com>> wrote: >>> > >> >> >>> > >> >> Michael, Max and other folks who are concerned about the >>> > compatibility with the apache release policy. Does the information >>> > in this thread sufficiently address your concerns? Especially the >>> > part where, the rc artifacts will be protected by a flag (i.e. >>> > --pre) from general consumption. >>> > >> >> >>> > >> >> On Tue, Apr 30, 2019 at 3:59 PM Robert Bradshaw >>> > <rober...@google.com <mailto:rober...@google.com>> wrote: >>> > >> >>> >>> > >> >>> On Tue, Apr 30, 2019 at 6:11 PM Ahmet Altay >>> > <al...@google.com <mailto:al...@google.com>> wrote: >>> > >> >>> > >>> > >> >>> > This conversation get quite Python centric. Is there a >>> > similar need for Java? >>> > >> >>> >>> > >> >>> I think Java is already covered. Go is a different story >>> > (but the even >>> > >> >>> versioning and releasing is being worked out). >>> > >> >>> >>> > >> >>> > On Tue, Apr 30, 2019 at 4:54 AM Robert Bradshaw >>> > <rober...@google.com <mailto:rober...@google.com>> wrote: >>> > >> >>> >> >>> > >> >>> >> If we can, by the apache guidelines, post RCs to pypy >>> that is >>> > >> >>> >> definitely the way to go. (Note that test.pypi is for >>> > developing >>> > >> >>> >> against the pypi interface, not for pushing anything >>> > real.) The caveat >>> > >> >>> >> about naming these with rcN in the version number still >>> > applies >>> > >> >>> >> (that's how pypi guards them against non-explicit >>> installs). >>> > >> >>> > >>> > >> >>> > Related to the caveat, I believe this can be easily >>> > scripted or even made part of the travis/wheels pipeline to take >>> the >>> > release branch, edit the version string in place to add rc, and >>> > build the necessary files. >>> > >> >>> >>> > >> >>> Yes. But the resulting artifacts would have to be rebuilt >>> (and >>> > >> >>> re-signed) without the version edit for the actual release. >>> > (Well, we >>> > >> >>> could possibly edit the artifacts rather than rebuild >>> them.) And >>> > >> >>> pushing un-edited ones early would be really bad. (It's the >>> > classic >>> > >> >>> tension of whether a pre-release should be marked >>> internally or >>> > >> >>> externally, re-publishing a new set of bits for the actual >>> > release or >>> > >> >>> re-using version numbers for different sets of bits. Pypi >>> > does one, >>> > >> >>> apache does another...) >>> > >> >>> >>> > >> >>> >> The advantage is that a user can do "pip install --pre >>> > apache-beam" to >>> > >> >>> >> get the latest rc rather than "pip install >>> > >> >>> >> >>> > >>> https://dist.apache.org/repos/dist/dev/beam/changing/and/ephemeral/path"; >>> > >> >>> >> >>> > >> >>> >> On Mon, Apr 29, 2019 at 11:34 PM Pablo Estrada >>> > <pabl...@google.com <mailto:pabl...@google.com>> wrote: >>> > >> >>> >> > >>> > >> >>> >> > Aw that's interesting! >>> > >> >>> >> > >>> > >> >>> >> > I think, with these considerations, I am only >>> > marginally more inclined towards publishing to test.pypi. That >>> would >>> > make me a +0.9 on publishing RCs to the main pip repo then. >>> > >> >>> >> > >>> > >> >>> >> > Thanks for doing the research Ahmet. :) >>> > >> >>> >> > Best >>> > >> >>> >> > -P >>> > >> >>> >> > >>> > >> >>> >> > On Mon, Apr 29, 2019 at 1:53 PM Ahmet Altay >>> > <al...@google.com <mailto:al...@google.com>> wrote: >>> > >> >>> >> >> >>> > >> >>> >> >> I asked to Airflow folks about this. See [1] for the >>> > full response and a link to one of their RC emails. To summarize >>> > their position (specifically for pypi) is: Unless a user does >>> > something explicit (such as using a flag, or explicitly requesting >>> > an rc release), pip install will not serve RC binaries. And that is >>> > compatible with RC section of >>> > http://www.apache.org/legal/release-policy.html#release-types >>> > >> >>> >> >> >>> > >> >>> >> >> Ahmet >>> > >> >>> >> >> >>> > >> >>> >> >> [1] >>> > >>> https://lists.apache.org/thread.html/f1f342332c1e180f57d60285bebe614ffa77bb53c4f74c4cbc049096@%3Cdev.airflow.apache.org%3E >>> > >> >>> >> >> >>> > >> >>> >> >> On Fri, Apr 26, 2019 at 3:38 PM Ahmet Altay >>> > <al...@google.com <mailto:al...@google.com>> wrote: >>> > >> >>> >> >>> >>> > >> >>> >> >>> The incremental value of publishing python artifacts >>> > to a separate place but not to actual pypi listing will be low. >>> > Users can already download RC artifacts, or even pip install from >>> > http location directly. I think the incremental value will be low, >>> > because for a user or a downstream library to test with Beam RCs >>> > using their usual ways will still require them to get other >>> > dependencies from the regular pypi listing. That would mean they >>> > need to change their setup to test with beam rcs, which is the same >>> > state as today. There will be some incremental value of putting >>> them >>> > in more obvious places (e.g. pypi test repository). I would rather >>> > not complicate the release process for doing this. >>> > >> >>> >> >>> >>> > >> >>> >> >>> >>> > >> >>> >> >>> >>> > >> >>> >> >>> On Thu, Apr 25, 2019 at 2:25 PM Kenneth Knowles >>> > <k...@apache.org <mailto:k...@apache.org>> wrote: >>> > >> >>> >> >>>> >>> > >> >>> >> >>>> Pip is also able to be pointed at any raw hosted >>> > directory for the install, right? So we could publish RCs or >>> > snapshots somewhere with more obvious caveats and not interfere >>> with >>> > the pypi list of actual releases. Much like the Java snapshots are >>> > stored in a separate opt-in repository. >>> > >> >>> >> >>>> >>> > >> >>> >> >>>> Kenn >>> > >> >>> >> >>>> >>> > >> >>> >> >>>> On Thu, Apr 25, 2019 at 5:39 AM Maximilian Michels >>> > <m...@apache.org <mailto:m...@apache.org>> wrote: >>> > >> >>> >> >>>>> >>> > >> >>> >> >>>>> > wouldn't that be in conflict with Apache release >>> > policy [1] ? >>> > >> >>> >> >>>>> > [1] >>> http://www.apache.org/legal/release-policy.html >>> > >> >>> >> >>>>> >>> > >> >>> >> >>>>> Indeed, advertising pre-release artifacts is >>> > against ASF rules. For >>> > >> >>> >> >>>>> example, Flink was asked to remove a link to the >>> > Maven snapshot >>> > >> >>> >> >>>>> repository from their download page. >>> > >> >>> >> >>>>> >>> > >> >>> >> >>>>> However, that does not mean we cannot publish >>> > Python artifacts. We just >>> > >> >>> >> >>>>> have to clearly mark them for developers only and >>> > not advertise them >>> > >> >>> >> >>>>> alongside with the official releases. >>> > >> >>> >> >>>>> >>> > >> >>> >> >>>>> -Max >>> > >> >>> >> >>>>> >>> > >> >>> >> >>>>> On 25.04.19 10:23, Robert Bradshaw wrote: >>> > >> >>> >> >>>>> > Don't we push java artifacts to maven >>> > repositories as part of the RC >>> > >> >>> >> >>>>> > process? And completely unvetted snapshots? (Or >>> > is this OK because >>> > >> >>> >> >>>>> > they are special opt-in apache-only ones?) >>> > >> >>> >> >>>>> > >>> > >> >>> >> >>>>> > I am generally in favor of the idea, but would >>> > like to avoid increased >>> > >> >>> >> >>>>> > toil on the release manager. >>> > >> >>> >> >>>>> > >>> > >> >>> >> >>>>> > One potential hitch I see is that current >>> release >>> > process updates the >>> > >> >>> >> >>>>> > versions to x.y.z (no RC or other pre-release >>> > indicator in the version >>> > >> >>> >> >>>>> > number) whereas pypi (and other systems) >>> > typically expect distinct >>> > >> >>> >> >>>>> > (recognizable) version numbers for each attempt, >>> > and only the actual >>> > >> >>> >> >>>>> > final result has the actual final release >>> version. >>> > >> >>> >> >>>>> > >>> > >> >>> >> >>>>> > On Thu, Apr 25, 2019 at 6:38 AM Ahmet Altay >>> > <al...@google.com <mailto:al...@google.com>> wrote: >>> > >> >>> >> >>>>> >> >>> > >> >>> >> >>>>> >> I do not know the answer.I believe this will be >>> > similar to sharing the RC artifacts for validation purposes and >>> > would not be a formal release by itself. But I am not an expert and >>> > I hope others will share their opinions. >>> > >> >>> >> >>>>> >> >>> > >> >>> >> >>>>> >> I quickly searched pypi for apache projects and >>> > found at least airflow [1] and libcloud [2] are publishing rc >>> > artifacts to pypi. We can reach out to those communities and learn >>> > about their processes. >>> > >> >>> >> >>>>> >> >>> > >> >>> >> >>>>> >> Ahmet >>> > >> >>> >> >>>>> >> >>> > >> >>> >> >>>>> >> [1] >>> https://pypi.org/project/apache-airflow/#history >>> > >> >>> >> >>>>> >> [2] >>> > https://pypi.org/project/apache-libcloud/#history >>> > >> >>> >> >>>>> >> >>> > >> >>> >> >>>>> >> On Wed, Apr 24, 2019 at 6:15 PM Michael Luckey >>> > <adude3...@gmail.com <mailto:adude3...@gmail.com>> wrote: >>> > >> >>> >> >>>>> >>> >>> > >> >>> >> >>>>> >>> Hi, >>> > >> >>> >> >>>>> >>> >>> > >> >>> >> >>>>> >>> wouldn't that be in conflict with Apache >>> > release policy [1] ? >>> > >> >>> >> >>>>> >>> >>> > >> >>> >> >>>>> >>> [1] >>> http://www.apache.org/legal/release-policy.html >>> > >> >>> >> >>>>> >>> >>> > >> >>> >> >>>>> >>> On Thu, Apr 25, 2019 at 1:35 AM Alan Myrvold >>> > <amyrv...@google.com <mailto:amyrv...@google.com>> wrote: >>> > >> >>> >> >>>>> >>>> >>> > >> >>> >> >>>>> >>>> Great idea. I like the RC candidates to >>> follow >>> > as much as the release artifact process as possible. >>> > >> >>> >> >>>>> >>>> >>> > >> >>> >> >>>>> >>>> On Wed, Apr 24, 2019 at 3:27 PM Ahmet Altay >>> > <al...@google.com <mailto:al...@google.com>> wrote: >>> > >> >>> >> >>>>> >>>>> >>> > >> >>> >> >>>>> >>>>> To clarify my proposal, I am proposing >>> > publishing to the production pypi repository with an rc tag in the >>> > version. And in turn allow users to depend on beam's rc version + >>> > all the other regular dependencies users would have directly from >>> pypi. >>> > >> >>> >> >>>>> >>>>> >>> > >> >>> >> >>>>> >>>>> Publishing to test pypi repo would also be >>> > helpful if test pypi repo also mirrors other packages that exist in >>> > the production pypi repository. >>> > >> >>> >> >>>>> >>>>> >>> > >> >>> >> >>>>> >>>>> On Wed, Apr 24, 2019 at 3:12 PM Pablo >>> Estrada >>> > <pabl...@google.com <mailto:pabl...@google.com>> wrote: >>> > >> >>> >> >>>>> >>>>>> >>> > >> >>> >> >>>>> >>>>>> I think this is a great idea. A way of >>> doing >>> > it for python would be by using the test repository for PyPi[1], >>> and >>> > that way we would not have to do an official PyPi release, but >>> still >>> > would be able to install it with pip (by passing an extra flag), >>> and >>> > test. >>> > >> >>> >> >>>>> >>>>>> >>> > >> >>> >> >>>>> >>>>>> In fact, there are some Beam artifacts >>> > already in there[2]. At some point I looked into this, but couldn't >>> > figure out who has access/the password for it. >>> > >> >>> >> >>>>> >>>>> >>> > >> >>> >> >>>>> >>>>> >>> > >> >>> >> >>>>> >>>>> I also don't know who owns beam package in >>> > test pypi repo. Does anybody know? >>> > >> >>> >> >>>>> >>>>> >>> > >> >>> >> >>>>> >>>>>> >>> > >> >>> >> >>>>> >>>>>> >>> > >> >>> >> >>>>> >>>>>> In short: +1, and I would suggest using the >>> > test PyPi repo to avoid publishing to the main PyPi repo. >>> > >> >>> >> >>>>> >>>>>> Best >>> > >> >>> >> >>>>> >>>>>> -P. >>> > >> >>> >> >>>>> >>>>>> >>> > >> >>> >> >>>>> >>>>>> [1] https://test.pypi.org/ >>> > >> >>> >> >>>>> >>>>>> [2] >>> https://test.pypi.org/project/apache-beam/ >>> > >> >>> >> >>>>> >>>>>> >>> > >> >>> >> >>>>> >>>>>> On Wed, Apr 24, 2019 at 3:04 PM Ahmet Altay >>> > <al...@google.com <mailto:al...@google.com>> wrote: >>> > >> >>> >> >>>>> >>>>>>> >>> > >> >>> >> >>>>> >>>>>>> Hi all, >>> > >> >>> >> >>>>> >>>>>>> >>> > >> >>> >> >>>>> >>>>>>> What do you think about the idea of >>> > publishing pre-release artifacts as part of the RC emails? >>> > >> >>> >> >>>>> >>>>>>> >>> > >> >>> >> >>>>> >>>>>>> For Python this would translate into >>> > publishing the same artifacts from RC email with a version like >>> > "2.X.0rcY" to pypi. I do not know, but I am guessing we can do a >>> > similar thing with Maven central for Java artifacts as well. >>> > >> >>> >> >>>>> >>>>>>> >>> > >> >>> >> >>>>> >>>>>>> Advantages would be: >>> > >> >>> >> >>>>> >>>>>>> - Allow end users to validate RCs for >>> their >>> > own purposes using the same exact process they will normally use. >>> > >> >>> >> >>>>> >>>>>>> - Enable early-adaptors to start using >>> RC >>> > releases early on in the release cycle if that is what they would >>> > like to do. This will in turn reduce time pressure on some >>> releases. >>> > Especially for cases like someone needs a release to be finalized >>> > for an upcoming event. >>> > >> >>> >> >>>>> >>>>>>> >>> > >> >>> >> >>>>> >>>>>>> There will also be disadvantages, some I >>> > could think of: >>> > >> >>> >> >>>>> >>>>>>> - Users could request support for RC >>> > artifacts. Hopefully in the form of feedback for us to improve the >>> > release. But it could also be in the form of folks using RC >>> > artifacts for production for a long time. >>> > >> >>> >> >>>>> >>>>>>> - It will add toil to the current release >>> > process, there will be one more step for each RC. I think for >>> python >>> > this will be a small step but nevertheless it will be additional >>> work. >>> > >> >>> >> >>>>> >>>>>>> >>> > >> >>> >> >>>>> >>>>>>> For an example of this, you can take a >>> look >>> > at tensorflow releases. For 1.13 there were 3 pre-releases [1]. >>> > >> >>> >> >>>>> >>>>>>> >>> > >> >>> >> >>>>> >>>>>>> Ahmet >>> > >> >>> >> >>>>> >>>>>>> >>> > >> >>> >> >>>>> >>>>>>> [1] >>> > https://pypi.org/project/tensorflow/#history >>> > >>> >>
