Thanks Ahmet for the time you put into this. AFAIU Roberts proposal resolves my concerns.
On Mon, May 6, 2019 at 8:01 PM Ahmet Altay <[email protected]> wrote: > Thank you Max. Michael, does the current state of the proposal address > your concerns? > > *From: *Maximilian Michels <[email protected]> > *Date: *Mon, May 6, 2019 at 8:32 AM > *To: * <[email protected]> > > Thanks for the link to the distribution guideline thread. The proposed >> process sounds good to me. >> >> Concerning Michael's point, I think we should follow the same procedure >> as we do for releasing to Maven Central, i.e. create a staging >> repository for validation which then gets released. >> >> The easiest way to do that, is to directly use the artifacts that we >> create during the release process: >> >> pip install https://dist.apache.org/repos/dist\ >> /dev/beam/<version>/python >> >> We wouldn't have to built a second artifact or change a version number. >> However, I understand if we want the additional convenience of: >> >> pip install --pre apache_beam >> >> As Robert mentioned, this would just be the additional step to rename >> the version number and push the RC package to PyPi. Not necessary in my >> opinion but I don't mind. >> > > Value proposition is for downstream library providers. They can update > their setup.py file to test beam rc in combination with rest of their other > dependencies and test that combination with ease. I agree it is not > necessary for us or any other party that just wants to test beam packages > and its direct dependencies in isolation. > > >> >> Thanks, >> Max >> >> On 02.05.19 19:43, Ahmet Altay wrote: >> > >> > >> > On Thu, May 2, 2019 at 9:29 AM Robert Bradshaw <[email protected] >> > <mailto:[email protected]>> wrote: >> > >> > On Thu, May 2, 2019 at 6:03 PM Michael Luckey <[email protected] >> > <mailto:[email protected]>> wrote: >> > > >> > > Yes, I understood this. But I m personally more paranoid about >> > releasing. >> > > >> > > So formally vote (and corresponding testing) was done on rc. If >> > we rebuild and resign, wouldn't that mean we also need to revote? >> > >> > Yeah, that's the sticking point. I suppose we could build the >> packages >> > with rc tags, push them to pypi, and also build them without rc >> tags, >> > and push those (and the full source tarball, which doesn't have an >> rc >> > tag either) to svn, and have the vote officially cover what's in svn >> > but the rc ones are just for convenience. >> > >> > >> > I prefer this way of double building. I think modifying the release >> > process as in your initial proposal would be risky. In that proposal, >> > what we test for validations would be different than what we push out >> as >> > final releases. The modification I see would be: >> > >> > 2c) Get the artifacts from svn, un-package, change the version to >> > include the rc number, and re-package and push. (This is very similar >> to >> > the process we use for building the wheel files today, with the >> > exception of changing the version part.) >> > >> > (But, given that I can "pip >> > install https::svn.apache.org/path/to/tarball >> > <http://svn.apache.org/path/to/tarball>" it'd primarily have >> > value for others doing "pip install --pre".) >> > >> > >> > This is regardless of whether is OK per apache to publish such >> binary >> > blobs to a third party place (though IMHO it follows the intent of >> the >> > release process). >> > >> > >> > This is also my understanding. What is being proposed here aligns with >> > what Kenn found out in the other threads. >> > >> > >> > > If I understand correctly, there will be some changed version >> > string in distributed sources (setup.py?). So there is some binary >> > difference. And just talking about me, doing that repackaging I >> > would certainly mess it up and package some unwanted changes. >> > >> > We definitely would not want this to be a manual step--I wouldn't >> > trust myself :). >> > >> > >> > > On Thu, May 2, 2019 at 5:43 PM Robert Bradshaw >> > <[email protected] <mailto:[email protected]>> wrote: >> > >> >> > >> On Thu, May 2, 2019 at 5:24 PM Michael Luckey >> > <[email protected] <mailto:[email protected]>> wrote: >> > >> > >> > >> > Thanks Ahmet for calling out to the airflow folks. I believe, >> > I am able to follow their argument. So from my point of view I do >> > not have an issue with apache policy. But honestly still trying to >> > wrap my head around Roberts concern with rebuilding/resigning. >> > Currently, our actual release is only a tag on source repo and >> > promoting artefacts. Do not yet understand how that needs to change >> > to get PyPi included. >> > >> >> > >> It's not a big change, but let me clarify. >> > >> >> > >> Currently our release preparation goes something like this: >> > >> >> > >> 1) Check out the repo, update the versions to 2.x, build and >> > sign the artifacts. >> > >> 2) Announce these artifacts as rcN >> > >> 2a) Push the artifacts to SVN dev/... >> > >> 2b) Push artifacts to the apache maven repository. >> > >> 3) Depending on vote, go back to step (1) or forward to step >> (4). >> > >> 4) Copy these artifacts as the actual release. >> > >> >> > >> Now if we just try to add (2c) Push these artifacts to Pypi, it >> will >> > >> be treated (by pypi's tooling, anyone who downloads the tarball, >> > ...) >> > >> as an actual release. You also can't re-push a tarball with the >> same >> > >> name and different contents (the idea being that named releases >> > should >> > >> never change). So we'd need to change step (1) to update the >> version >> > >> to 2.x.rcN *and* add a step in (4) to update the version to 2.x >> > (no rc >> > >> suffix), rebuild, resign before publishing. >> > >> >> > >> As mentioned, possibly the rcN suffix could be part of the >> building >> > >> step for Python. >> > >> >> > >> > On Wed, May 1, 2019 at 1:33 AM Ahmet Altay <[email protected] >> > <mailto:[email protected]>> wrote: >> > >> >> >> > >> >> Michael, Max and other folks who are concerned about the >> > compatibility with the apache release policy. Does the information >> > in this thread sufficiently address your concerns? Especially the >> > part where, the rc artifacts will be protected by a flag (i.e. >> > --pre) from general consumption. >> > >> >> >> > >> >> On Tue, Apr 30, 2019 at 3:59 PM Robert Bradshaw >> > <[email protected] <mailto:[email protected]>> wrote: >> > >> >>> >> > >> >>> On Tue, Apr 30, 2019 at 6:11 PM Ahmet Altay >> > <[email protected] <mailto:[email protected]>> wrote: >> > >> >>> > >> > >> >>> > This conversation get quite Python centric. Is there a >> > similar need for Java? >> > >> >>> >> > >> >>> I think Java is already covered. Go is a different story >> > (but the even >> > >> >>> versioning and releasing is being worked out). >> > >> >>> >> > >> >>> > On Tue, Apr 30, 2019 at 4:54 AM Robert Bradshaw >> > <[email protected] <mailto:[email protected]>> wrote: >> > >> >>> >> >> > >> >>> >> If we can, by the apache guidelines, post RCs to pypy >> that is >> > >> >>> >> definitely the way to go. (Note that test.pypi is for >> > developing >> > >> >>> >> against the pypi interface, not for pushing anything >> > real.) The caveat >> > >> >>> >> about naming these with rcN in the version number still >> > applies >> > >> >>> >> (that's how pypi guards them against non-explicit >> installs). >> > >> >>> > >> > >> >>> > Related to the caveat, I believe this can be easily >> > scripted or even made part of the travis/wheels pipeline to take the >> > release branch, edit the version string in place to add rc, and >> > build the necessary files. >> > >> >>> >> > >> >>> Yes. But the resulting artifacts would have to be rebuilt >> (and >> > >> >>> re-signed) without the version edit for the actual release. >> > (Well, we >> > >> >>> could possibly edit the artifacts rather than rebuild >> them.) And >> > >> >>> pushing un-edited ones early would be really bad. (It's the >> > classic >> > >> >>> tension of whether a pre-release should be marked >> internally or >> > >> >>> externally, re-publishing a new set of bits for the actual >> > release or >> > >> >>> re-using version numbers for different sets of bits. Pypi >> > does one, >> > >> >>> apache does another...) >> > >> >>> >> > >> >>> >> The advantage is that a user can do "pip install --pre >> > apache-beam" to >> > >> >>> >> get the latest rc rather than "pip install >> > >> >>> >> >> > >> https://dist.apache.org/repos/dist/dev/beam/changing/and/ephemeral/path"; >> > >> >>> >> >> > >> >>> >> On Mon, Apr 29, 2019 at 11:34 PM Pablo Estrada >> > <[email protected] <mailto:[email protected]>> wrote: >> > >> >>> >> > >> > >> >>> >> > Aw that's interesting! >> > >> >>> >> > >> > >> >>> >> > I think, with these considerations, I am only >> > marginally more inclined towards publishing to test.pypi. That would >> > make me a +0.9 on publishing RCs to the main pip repo then. >> > >> >>> >> > >> > >> >>> >> > Thanks for doing the research Ahmet. :) >> > >> >>> >> > Best >> > >> >>> >> > -P >> > >> >>> >> > >> > >> >>> >> > On Mon, Apr 29, 2019 at 1:53 PM Ahmet Altay >> > <[email protected] <mailto:[email protected]>> wrote: >> > >> >>> >> >> >> > >> >>> >> >> I asked to Airflow folks about this. See [1] for the >> > full response and a link to one of their RC emails. To summarize >> > their position (specifically for pypi) is: Unless a user does >> > something explicit (such as using a flag, or explicitly requesting >> > an rc release), pip install will not serve RC binaries. And that is >> > compatible with RC section of >> > http://www.apache.org/legal/release-policy.html#release-types >> > >> >>> >> >> >> > >> >>> >> >> Ahmet >> > >> >>> >> >> >> > >> >>> >> >> [1] >> > >> https://lists.apache.org/thread.html/f1f342332c1e180f57d60285bebe614ffa77bb53c4f74c4cbc049096@%3Cdev.airflow.apache.org%3E >> > >> >>> >> >> >> > >> >>> >> >> On Fri, Apr 26, 2019 at 3:38 PM Ahmet Altay >> > <[email protected] <mailto:[email protected]>> wrote: >> > >> >>> >> >>> >> > >> >>> >> >>> The incremental value of publishing python artifacts >> > to a separate place but not to actual pypi listing will be low. >> > Users can already download RC artifacts, or even pip install from >> > http location directly. I think the incremental value will be low, >> > because for a user or a downstream library to test with Beam RCs >> > using their usual ways will still require them to get other >> > dependencies from the regular pypi listing. That would mean they >> > need to change their setup to test with beam rcs, which is the same >> > state as today. There will be some incremental value of putting them >> > in more obvious places (e.g. pypi test repository). I would rather >> > not complicate the release process for doing this. >> > >> >>> >> >>> >> > >> >>> >> >>> >> > >> >>> >> >>> >> > >> >>> >> >>> On Thu, Apr 25, 2019 at 2:25 PM Kenneth Knowles >> > <[email protected] <mailto:[email protected]>> wrote: >> > >> >>> >> >>>> >> > >> >>> >> >>>> Pip is also able to be pointed at any raw hosted >> > directory for the install, right? So we could publish RCs or >> > snapshots somewhere with more obvious caveats and not interfere with >> > the pypi list of actual releases. Much like the Java snapshots are >> > stored in a separate opt-in repository. >> > >> >>> >> >>>> >> > >> >>> >> >>>> Kenn >> > >> >>> >> >>>> >> > >> >>> >> >>>> On Thu, Apr 25, 2019 at 5:39 AM Maximilian Michels >> > <[email protected] <mailto:[email protected]>> wrote: >> > >> >>> >> >>>>> >> > >> >>> >> >>>>> > wouldn't that be in conflict with Apache release >> > policy [1] ? >> > >> >>> >> >>>>> > [1] >> http://www.apache.org/legal/release-policy.html >> > >> >>> >> >>>>> >> > >> >>> >> >>>>> Indeed, advertising pre-release artifacts is >> > against ASF rules. For >> > >> >>> >> >>>>> example, Flink was asked to remove a link to the >> > Maven snapshot >> > >> >>> >> >>>>> repository from their download page. >> > >> >>> >> >>>>> >> > >> >>> >> >>>>> However, that does not mean we cannot publish >> > Python artifacts. We just >> > >> >>> >> >>>>> have to clearly mark them for developers only and >> > not advertise them >> > >> >>> >> >>>>> alongside with the official releases. >> > >> >>> >> >>>>> >> > >> >>> >> >>>>> -Max >> > >> >>> >> >>>>> >> > >> >>> >> >>>>> On 25.04.19 10:23, Robert Bradshaw wrote: >> > >> >>> >> >>>>> > Don't we push java artifacts to maven >> > repositories as part of the RC >> > >> >>> >> >>>>> > process? And completely unvetted snapshots? (Or >> > is this OK because >> > >> >>> >> >>>>> > they are special opt-in apache-only ones?) >> > >> >>> >> >>>>> > >> > >> >>> >> >>>>> > I am generally in favor of the idea, but would >> > like to avoid increased >> > >> >>> >> >>>>> > toil on the release manager. >> > >> >>> >> >>>>> > >> > >> >>> >> >>>>> > One potential hitch I see is that current release >> > process updates the >> > >> >>> >> >>>>> > versions to x.y.z (no RC or other pre-release >> > indicator in the version >> > >> >>> >> >>>>> > number) whereas pypi (and other systems) >> > typically expect distinct >> > >> >>> >> >>>>> > (recognizable) version numbers for each attempt, >> > and only the actual >> > >> >>> >> >>>>> > final result has the actual final release >> version. >> > >> >>> >> >>>>> > >> > >> >>> >> >>>>> > On Thu, Apr 25, 2019 at 6:38 AM Ahmet Altay >> > <[email protected] <mailto:[email protected]>> wrote: >> > >> >>> >> >>>>> >> >> > >> >>> >> >>>>> >> I do not know the answer.I believe this will be >> > similar to sharing the RC artifacts for validation purposes and >> > would not be a formal release by itself. But I am not an expert and >> > I hope others will share their opinions. >> > >> >>> >> >>>>> >> >> > >> >>> >> >>>>> >> I quickly searched pypi for apache projects and >> > found at least airflow [1] and libcloud [2] are publishing rc >> > artifacts to pypi. We can reach out to those communities and learn >> > about their processes. >> > >> >>> >> >>>>> >> >> > >> >>> >> >>>>> >> Ahmet >> > >> >>> >> >>>>> >> >> > >> >>> >> >>>>> >> [1] >> https://pypi.org/project/apache-airflow/#history >> > >> >>> >> >>>>> >> [2] >> > https://pypi.org/project/apache-libcloud/#history >> > >> >>> >> >>>>> >> >> > >> >>> >> >>>>> >> On Wed, Apr 24, 2019 at 6:15 PM Michael Luckey >> > <[email protected] <mailto:[email protected]>> wrote: >> > >> >>> >> >>>>> >>> >> > >> >>> >> >>>>> >>> Hi, >> > >> >>> >> >>>>> >>> >> > >> >>> >> >>>>> >>> wouldn't that be in conflict with Apache >> > release policy [1] ? >> > >> >>> >> >>>>> >>> >> > >> >>> >> >>>>> >>> [1] >> http://www.apache.org/legal/release-policy.html >> > >> >>> >> >>>>> >>> >> > >> >>> >> >>>>> >>> On Thu, Apr 25, 2019 at 1:35 AM Alan Myrvold >> > <[email protected] <mailto:[email protected]>> wrote: >> > >> >>> >> >>>>> >>>> >> > >> >>> >> >>>>> >>>> Great idea. I like the RC candidates to follow >> > as much as the release artifact process as possible. >> > >> >>> >> >>>>> >>>> >> > >> >>> >> >>>>> >>>> On Wed, Apr 24, 2019 at 3:27 PM Ahmet Altay >> > <[email protected] <mailto:[email protected]>> wrote: >> > >> >>> >> >>>>> >>>>> >> > >> >>> >> >>>>> >>>>> To clarify my proposal, I am proposing >> > publishing to the production pypi repository with an rc tag in the >> > version. And in turn allow users to depend on beam's rc version + >> > all the other regular dependencies users would have directly from >> pypi. >> > >> >>> >> >>>>> >>>>> >> > >> >>> >> >>>>> >>>>> Publishing to test pypi repo would also be >> > helpful if test pypi repo also mirrors other packages that exist in >> > the production pypi repository. >> > >> >>> >> >>>>> >>>>> >> > >> >>> >> >>>>> >>>>> On Wed, Apr 24, 2019 at 3:12 PM Pablo Estrada >> > <[email protected] <mailto:[email protected]>> wrote: >> > >> >>> >> >>>>> >>>>>> >> > >> >>> >> >>>>> >>>>>> I think this is a great idea. A way of doing >> > it for python would be by using the test repository for PyPi[1], and >> > that way we would not have to do an official PyPi release, but still >> > would be able to install it with pip (by passing an extra flag), and >> > test. >> > >> >>> >> >>>>> >>>>>> >> > >> >>> >> >>>>> >>>>>> In fact, there are some Beam artifacts >> > already in there[2]. At some point I looked into this, but couldn't >> > figure out who has access/the password for it. >> > >> >>> >> >>>>> >>>>> >> > >> >>> >> >>>>> >>>>> >> > >> >>> >> >>>>> >>>>> I also don't know who owns beam package in >> > test pypi repo. Does anybody know? >> > >> >>> >> >>>>> >>>>> >> > >> >>> >> >>>>> >>>>>> >> > >> >>> >> >>>>> >>>>>> >> > >> >>> >> >>>>> >>>>>> In short: +1, and I would suggest using the >> > test PyPi repo to avoid publishing to the main PyPi repo. >> > >> >>> >> >>>>> >>>>>> Best >> > >> >>> >> >>>>> >>>>>> -P. >> > >> >>> >> >>>>> >>>>>> >> > >> >>> >> >>>>> >>>>>> [1] https://test.pypi.org/ >> > >> >>> >> >>>>> >>>>>> [2] >> https://test.pypi.org/project/apache-beam/ >> > >> >>> >> >>>>> >>>>>> >> > >> >>> >> >>>>> >>>>>> On Wed, Apr 24, 2019 at 3:04 PM Ahmet Altay >> > <[email protected] <mailto:[email protected]>> wrote: >> > >> >>> >> >>>>> >>>>>>> >> > >> >>> >> >>>>> >>>>>>> Hi all, >> > >> >>> >> >>>>> >>>>>>> >> > >> >>> >> >>>>> >>>>>>> What do you think about the idea of >> > publishing pre-release artifacts as part of the RC emails? >> > >> >>> >> >>>>> >>>>>>> >> > >> >>> >> >>>>> >>>>>>> For Python this would translate into >> > publishing the same artifacts from RC email with a version like >> > "2.X.0rcY" to pypi. I do not know, but I am guessing we can do a >> > similar thing with Maven central for Java artifacts as well. >> > >> >>> >> >>>>> >>>>>>> >> > >> >>> >> >>>>> >>>>>>> Advantages would be: >> > >> >>> >> >>>>> >>>>>>> - Allow end users to validate RCs for their >> > own purposes using the same exact process they will normally use. >> > >> >>> >> >>>>> >>>>>>> - Enable early-adaptors to start using RC >> > releases early on in the release cycle if that is what they would >> > like to do. This will in turn reduce time pressure on some releases. >> > Especially for cases like someone needs a release to be finalized >> > for an upcoming event. >> > >> >>> >> >>>>> >>>>>>> >> > >> >>> >> >>>>> >>>>>>> There will also be disadvantages, some I >> > could think of: >> > >> >>> >> >>>>> >>>>>>> - Users could request support for RC >> > artifacts. Hopefully in the form of feedback for us to improve the >> > release. But it could also be in the form of folks using RC >> > artifacts for production for a long time. >> > >> >>> >> >>>>> >>>>>>> - It will add toil to the current release >> > process, there will be one more step for each RC. I think for python >> > this will be a small step but nevertheless it will be additional >> work. >> > >> >>> >> >>>>> >>>>>>> >> > >> >>> >> >>>>> >>>>>>> For an example of this, you can take a look >> > at tensorflow releases. For 1.13 there were 3 pre-releases [1]. >> > >> >>> >> >>>>> >>>>>>> >> > >> >>> >> >>>>> >>>>>>> Ahmet >> > >> >>> >> >>>>> >>>>>>> >> > >> >>> >> >>>>> >>>>>>> [1] >> > https://pypi.org/project/tensorflow/#history >> > >> >
