On Thu, May 2, 2019 at 8:56 AM Kenneth Knowles <[email protected]> wrote:
> Pulling out the relevant pypi bit w.r.t. RCs: > > >> - Release candidates, nightly or snapshots need to be clearly tagged as >> pre-release on https://pypi.org/project/apache<project>/#history >> - The latest version should not point to an artefact containing >> unapproved code e.g. to a release candidate or snapshot >> > > These guidelines were pending assessment by legal & infra. I don't know if > there has been an update. It has been a few months. > Thanks a lot for finding all these information. IMO, this proposal fits with the above guideline. Where can we track/ask about the information you found specifically about pypi? > > Kenn > > On Thu, May 2, 2019 at 8:51 AM Kenneth Knowles <[email protected]> wrote: > >> Ah, and here's one on general@incubator specifically about RCs: >> https://lists.apache.org/thread.html/c4afcf0807d71f844d912a7e5fe6b481f0779bdcf88ccf9abe50a160@%3Cgeneral.incubator.apache.org%3E >> >> Kenn >> >> On Thu, May 2, 2019 at 8:49 AM Kenneth Knowles <[email protected]> wrote: >> >>> I'd suggest looking for experience beyond Beam and Airflow. I don't see >>> links to some relevant threads. >>> >>> Here's one from legal-discuss@ about binary channels and how they >>> relate to source releases: >>> https://lists.apache.org/thread.html/d578819f1afa6b8fb697ea72083e0fb05e43938a23d6e7bb804069b8@%3Clegal-discuss.apache.org%3E >>> >>> Here's one from the incubator about how RCs and tags relate to ASF >>> release policy: >>> https://lists.apache.org/thread.html/982077ef279e50b260d302d96685e40be6fcabdb0bd43d519621cf27@%3Cgeneral.incubator.apache.org%3E >>> >>> (apologies if these are duplicate links - I re-scanned the thread and >>> did not spot them) >>> >>> Kenn >>> >>> On Thu, May 2, 2019 at 8:43 AM Robert Bradshaw <[email protected]> >>> wrote: >>> >>>> On Thu, May 2, 2019 at 5:24 PM Michael Luckey <[email protected]> >>>> wrote: >>>> > >>>> > Thanks Ahmet for calling out to the airflow folks. I believe, I am >>>> able to follow their argument. So from my point of view I do not have an >>>> issue with apache policy. But honestly still trying to wrap my head around >>>> Roberts concern with rebuilding/resigning. Currently, our actual release is >>>> only a tag on source repo and promoting artefacts. Do not yet understand >>>> how that needs to change to get PyPi included. >>>> >>>> It's not a big change, but let me clarify. >>>> >>>> Currently our release preparation goes something like this: >>>> >>>> 1) Check out the repo, update the versions to 2.x, build and sign the >>>> artifacts. >>>> 2) Announce these artifacts as rcN >>>> 2a) Push the artifacts to SVN dev/... >>>> 2b) Push artifacts to the apache maven repository. >>>> 3) Depending on vote, go back to step (1) or forward to step (4). >>>> 4) Copy these artifacts as the actual release. >>>> >>>> Now if we just try to add (2c) Push these artifacts to Pypi, it will >>>> be treated (by pypi's tooling, anyone who downloads the tarball, ...) >>>> as an actual release. You also can't re-push a tarball with the same >>>> name and different contents (the idea being that named releases should >>>> never change). So we'd need to change step (1) to update the version >>>> to 2.x.rcN *and* add a step in (4) to update the version to 2.x (no rc >>>> suffix), rebuild, resign before publishing. >>>> >>>> As mentioned, possibly the rcN suffix could be part of the building >>>> step for Python. >>>> >>>> > On Wed, May 1, 2019 at 1:33 AM Ahmet Altay <[email protected]> wrote: >>>> >> >>>> >> Michael, Max and other folks who are concerned about the >>>> compatibility with the apache release policy. Does the information in this >>>> thread sufficiently address your concerns? Especially the part where, the >>>> rc artifacts will be protected by a flag (i.e. --pre) from general >>>> consumption. >>>> >> >>>> >> On Tue, Apr 30, 2019 at 3:59 PM Robert Bradshaw <[email protected]> >>>> wrote: >>>> >>> >>>> >>> On Tue, Apr 30, 2019 at 6:11 PM Ahmet Altay <[email protected]> >>>> wrote: >>>> >>> > >>>> >>> > This conversation get quite Python centric. Is there a similar >>>> need for Java? >>>> >>> >>>> >>> I think Java is already covered. Go is a different story (but the >>>> even >>>> >>> versioning and releasing is being worked out). >>>> >>> >>>> >>> > On Tue, Apr 30, 2019 at 4:54 AM Robert Bradshaw < >>>> [email protected]> wrote: >>>> >>> >> >>>> >>> >> If we can, by the apache guidelines, post RCs to pypy that is >>>> >>> >> definitely the way to go. (Note that test.pypi is for developing >>>> >>> >> against the pypi interface, not for pushing anything real.) The >>>> caveat >>>> >>> >> about naming these with rcN in the version number still applies >>>> >>> >> (that's how pypi guards them against non-explicit installs). >>>> >>> > >>>> >>> > Related to the caveat, I believe this can be easily scripted or >>>> even made part of the travis/wheels pipeline to take the release branch, >>>> edit the version string in place to add rc, and build the necessary files. >>>> >>> >>>> >>> Yes. But the resulting artifacts would have to be rebuilt (and >>>> >>> re-signed) without the version edit for the actual release. (Well, >>>> we >>>> >>> could possibly edit the artifacts rather than rebuild them.) And >>>> >>> pushing un-edited ones early would be really bad. (It's the classic >>>> >>> tension of whether a pre-release should be marked internally or >>>> >>> externally, re-publishing a new set of bits for the actual release >>>> or >>>> >>> re-using version numbers for different sets of bits. Pypi does one, >>>> >>> apache does another...) >>>> >>> >>>> >>> >> The advantage is that a user can do "pip install --pre >>>> apache-beam" to >>>> >>> >> get the latest rc rather than "pip install >>>> >>> >> >>>> https://dist.apache.org/repos/dist/dev/beam/changing/and/ephemeral/path >>>> " >>>> >>> >> >>>> >>> >> On Mon, Apr 29, 2019 at 11:34 PM Pablo Estrada < >>>> [email protected]> wrote: >>>> >>> >> > >>>> >>> >> > Aw that's interesting! >>>> >>> >> > >>>> >>> >> > I think, with these considerations, I am only marginally more >>>> inclined towards publishing to test.pypi. That would make me a +0.9 on >>>> publishing RCs to the main pip repo then. >>>> >>> >> > >>>> >>> >> > Thanks for doing the research Ahmet. :) >>>> >>> >> > Best >>>> >>> >> > -P >>>> >>> >> > >>>> >>> >> > On Mon, Apr 29, 2019 at 1:53 PM Ahmet Altay <[email protected]> >>>> wrote: >>>> >>> >> >> >>>> >>> >> >> I asked to Airflow folks about this. See [1] for the full >>>> response and a link to one of their RC emails. To summarize their position >>>> (specifically for pypi) is: Unless a user does something explicit (such as >>>> using a flag, or explicitly requesting an rc release), pip install will not >>>> serve RC binaries. And that is compatible with RC section of >>>> http://www.apache.org/legal/release-policy.html#release-types >>>> >>> >> >> >>>> >>> >> >> Ahmet >>>> >>> >> >> >>>> >>> >> >> [1] >>>> https://lists.apache.org/thread.html/f1f342332c1e180f57d60285bebe614ffa77bb53c4f74c4cbc049096@%3Cdev.airflow.apache.org%3E >>>> >>> >> >> >>>> >>> >> >> On Fri, Apr 26, 2019 at 3:38 PM Ahmet Altay <[email protected]> >>>> wrote: >>>> >>> >> >>> >>>> >>> >> >>> The incremental value of publishing python artifacts to a >>>> separate place but not to actual pypi listing will be low. Users can >>>> already download RC artifacts, or even pip install from http location >>>> directly. I think the incremental value will be low, because for a user or >>>> a downstream library to test with Beam RCs using their usual ways will >>>> still require them to get other dependencies from the regular pypi listing. >>>> That would mean they need to change their setup to test with beam rcs, >>>> which is the same state as today. There will be some incremental value of >>>> putting them in more obvious places (e.g. pypi test repository). I would >>>> rather not complicate the release process for doing this. >>>> >>> >> >>> >>>> >>> >> >>> >>>> >>> >> >>> >>>> >>> >> >>> On Thu, Apr 25, 2019 at 2:25 PM Kenneth Knowles < >>>> [email protected]> wrote: >>>> >>> >> >>>> >>>> >>> >> >>>> Pip is also able to be pointed at any raw hosted directory >>>> for the install, right? So we could publish RCs or snapshots somewhere with >>>> more obvious caveats and not interfere with the pypi list of actual >>>> releases. Much like the Java snapshots are stored in a separate opt-in >>>> repository. >>>> >>> >> >>>> >>>> >>> >> >>>> Kenn >>>> >>> >> >>>> >>>> >>> >> >>>> On Thu, Apr 25, 2019 at 5:39 AM Maximilian Michels < >>>> [email protected]> wrote: >>>> >>> >> >>>>> >>>> >>> >> >>>>> > wouldn't that be in conflict with Apache release policy >>>> [1] ? >>>> >>> >> >>>>> > [1] http://www.apache.org/legal/release-policy.html >>>> >>> >> >>>>> >>>> >>> >> >>>>> Indeed, advertising pre-release artifacts is against ASF >>>> rules. For >>>> >>> >> >>>>> example, Flink was asked to remove a link to the Maven >>>> snapshot >>>> >>> >> >>>>> repository from their download page. >>>> >>> >> >>>>> >>>> >>> >> >>>>> However, that does not mean we cannot publish Python >>>> artifacts. We just >>>> >>> >> >>>>> have to clearly mark them for developers only and not >>>> advertise them >>>> >>> >> >>>>> alongside with the official releases. >>>> >>> >> >>>>> >>>> >>> >> >>>>> -Max >>>> >>> >> >>>>> >>>> >>> >> >>>>> On 25.04.19 10:23, Robert Bradshaw wrote: >>>> >>> >> >>>>> > Don't we push java artifacts to maven repositories as >>>> part of the RC >>>> >>> >> >>>>> > process? And completely unvetted snapshots? (Or is this >>>> OK because >>>> >>> >> >>>>> > they are special opt-in apache-only ones?) >>>> >>> >> >>>>> > >>>> >>> >> >>>>> > I am generally in favor of the idea, but would like to >>>> avoid increased >>>> >>> >> >>>>> > toil on the release manager. >>>> >>> >> >>>>> > >>>> >>> >> >>>>> > One potential hitch I see is that current release >>>> process updates the >>>> >>> >> >>>>> > versions to x.y.z (no RC or other pre-release indicator >>>> in the version >>>> >>> >> >>>>> > number) whereas pypi (and other systems) typically >>>> expect distinct >>>> >>> >> >>>>> > (recognizable) version numbers for each attempt, and >>>> only the actual >>>> >>> >> >>>>> > final result has the actual final release version. >>>> >>> >> >>>>> > >>>> >>> >> >>>>> > On Thu, Apr 25, 2019 at 6:38 AM Ahmet Altay < >>>> [email protected]> wrote: >>>> >>> >> >>>>> >> >>>> >>> >> >>>>> >> I do not know the answer.I believe this will be similar >>>> to sharing the RC artifacts for validation purposes and would not be a >>>> formal release by itself. But I am not an expert and I hope others will >>>> share their opinions. >>>> >>> >> >>>>> >> >>>> >>> >> >>>>> >> I quickly searched pypi for apache projects and found >>>> at least airflow [1] and libcloud [2] are publishing rc artifacts to pypi. >>>> We can reach out to those communities and learn about their processes. >>>> >>> >> >>>>> >> >>>> >>> >> >>>>> >> Ahmet >>>> >>> >> >>>>> >> >>>> >>> >> >>>>> >> [1] https://pypi.org/project/apache-airflow/#history >>>> >>> >> >>>>> >> [2] https://pypi.org/project/apache-libcloud/#history >>>> >>> >> >>>>> >> >>>> >>> >> >>>>> >> On Wed, Apr 24, 2019 at 6:15 PM Michael Luckey < >>>> [email protected]> wrote: >>>> >>> >> >>>>> >>> >>>> >>> >> >>>>> >>> Hi, >>>> >>> >> >>>>> >>> >>>> >>> >> >>>>> >>> wouldn't that be in conflict with Apache release >>>> policy [1] ? >>>> >>> >> >>>>> >>> >>>> >>> >> >>>>> >>> [1] http://www.apache.org/legal/release-policy.html >>>> >>> >> >>>>> >>> >>>> >>> >> >>>>> >>> On Thu, Apr 25, 2019 at 1:35 AM Alan Myrvold < >>>> [email protected]> wrote: >>>> >>> >> >>>>> >>>> >>>> >>> >> >>>>> >>>> Great idea. I like the RC candidates to follow as >>>> much as the release artifact process as possible. >>>> >>> >> >>>>> >>>> >>>> >>> >> >>>>> >>>> On Wed, Apr 24, 2019 at 3:27 PM Ahmet Altay < >>>> [email protected]> wrote: >>>> >>> >> >>>>> >>>>> >>>> >>> >> >>>>> >>>>> To clarify my proposal, I am proposing publishing to >>>> the production pypi repository with an rc tag in the version. And in turn >>>> allow users to depend on beam's rc version + all the other regular >>>> dependencies users would have directly from pypi. >>>> >>> >> >>>>> >>>>> >>>> >>> >> >>>>> >>>>> Publishing to test pypi repo would also be helpful >>>> if test pypi repo also mirrors other packages that exist in the production >>>> pypi repository. >>>> >>> >> >>>>> >>>>> >>>> >>> >> >>>>> >>>>> On Wed, Apr 24, 2019 at 3:12 PM Pablo Estrada < >>>> [email protected]> wrote: >>>> >>> >> >>>>> >>>>>> >>>> >>> >> >>>>> >>>>>> I think this is a great idea. A way of doing it for >>>> python would be by using the test repository for PyPi[1], and that way we >>>> would not have to do an official PyPi release, but still would be able to >>>> install it with pip (by passing an extra flag), and test. >>>> >>> >> >>>>> >>>>>> >>>> >>> >> >>>>> >>>>>> In fact, there are some Beam artifacts already in >>>> there[2]. At some point I looked into this, but couldn't figure out who has >>>> access/the password for it. >>>> >>> >> >>>>> >>>>> >>>> >>> >> >>>>> >>>>> >>>> >>> >> >>>>> >>>>> I also don't know who owns beam package in test pypi >>>> repo. Does anybody know? >>>> >>> >> >>>>> >>>>> >>>> >>> >> >>>>> >>>>>> >>>> >>> >> >>>>> >>>>>> >>>> >>> >> >>>>> >>>>>> In short: +1, and I would suggest using the test >>>> PyPi repo to avoid publishing to the main PyPi repo. >>>> >>> >> >>>>> >>>>>> Best >>>> >>> >> >>>>> >>>>>> -P. >>>> >>> >> >>>>> >>>>>> >>>> >>> >> >>>>> >>>>>> [1] https://test.pypi.org/ >>>> >>> >> >>>>> >>>>>> [2] https://test.pypi.org/project/apache-beam/ >>>> >>> >> >>>>> >>>>>> >>>> >>> >> >>>>> >>>>>> On Wed, Apr 24, 2019 at 3:04 PM Ahmet Altay < >>>> [email protected]> wrote: >>>> >>> >> >>>>> >>>>>>> >>>> >>> >> >>>>> >>>>>>> Hi all, >>>> >>> >> >>>>> >>>>>>> >>>> >>> >> >>>>> >>>>>>> What do you think about the idea of publishing >>>> pre-release artifacts as part of the RC emails? >>>> >>> >> >>>>> >>>>>>> >>>> >>> >> >>>>> >>>>>>> For Python this would translate into publishing >>>> the same artifacts from RC email with a version like "2.X.0rcY" to pypi. I >>>> do not know, but I am guessing we can do a similar thing with Maven central >>>> for Java artifacts as well. >>>> >>> >> >>>>> >>>>>>> >>>> >>> >> >>>>> >>>>>>> Advantages would be: >>>> >>> >> >>>>> >>>>>>> - Allow end users to validate RCs for their own >>>> purposes using the same exact process they will normally use. >>>> >>> >> >>>>> >>>>>>> - Enable early-adaptors to start using RC >>>> releases early on in the release cycle if that is what they would like to >>>> do. This will in turn reduce time pressure on some releases. Especially for >>>> cases like someone needs a release to be finalized for an upcoming event. >>>> >>> >> >>>>> >>>>>>> >>>> >>> >> >>>>> >>>>>>> There will also be disadvantages, some I could >>>> think of: >>>> >>> >> >>>>> >>>>>>> - Users could request support for RC artifacts. >>>> Hopefully in the form of feedback for us to improve the release. But it >>>> could also be in the form of folks using RC artifacts for production for a >>>> long time. >>>> >>> >> >>>>> >>>>>>> - It will add toil to the current release process, >>>> there will be one more step for each RC. I think for python this will be a >>>> small step but nevertheless it will be additional work. >>>> >>> >> >>>>> >>>>>>> >>>> >>> >> >>>>> >>>>>>> For an example of this, you can take a look at >>>> tensorflow releases. For 1.13 there were 3 pre-releases [1]. >>>> >>> >> >>>>> >>>>>>> >>>> >>> >> >>>>> >>>>>>> Ahmet >>>> >>> >> >>>>> >>>>>>> >>>> >>> >> >>>>> >>>>>>> [1] https://pypi.org/project/tensorflow/#history >>>> >>>
