On Thu, May 2, 2019 at 9:29 AM Robert Bradshaw <[email protected]
<mailto:[email protected]>> wrote:
On Thu, May 2, 2019 at 6:03 PM Michael Luckey <[email protected]
<mailto:[email protected]>> wrote:
>
> Yes, I understood this. But I m personally more paranoid about
releasing.
>
> So formally vote (and corresponding testing) was done on rc. If
we rebuild and resign, wouldn't that mean we also need to revote?
Yeah, that's the sticking point. I suppose we could build the packages
with rc tags, push them to pypi, and also build them without rc tags,
and push those (and the full source tarball, which doesn't have an rc
tag either) to svn, and have the vote officially cover what's in svn
but the rc ones are just for convenience.
I prefer this way of double building. I think modifying the release
process as in your initial proposal would be risky. In that proposal,
what we test for validations would be different than what we push out as
final releases. The modification I see would be:
2c) Get the artifacts from svn, un-package, change the version to
include the rc number, and re-package and push. (This is very similar to
the process we use for building the wheel files today, with the
exception of changing the version part.)
(But, given that I can "pip
install https::svn.apache.org/path/to/tarball
<http://svn.apache.org/path/to/tarball>" it'd primarily have
value for others doing "pip install --pre".)
This is regardless of whether is OK per apache to publish such binary
blobs to a third party place (though IMHO it follows the intent of the
release process).
This is also my understanding. What is being proposed here aligns with
what Kenn found out in the other threads.
> If I understand correctly, there will be some changed version
string in distributed sources (setup.py?). So there is some binary
difference. And just talking about me, doing that repackaging I
would certainly mess it up and package some unwanted changes.
We definitely would not want this to be a manual step--I wouldn't
trust myself :).
> On Thu, May 2, 2019 at 5:43 PM Robert Bradshaw
<[email protected] <mailto:[email protected]>> wrote:
>>
>> On Thu, May 2, 2019 at 5:24 PM Michael Luckey
<[email protected] <mailto:[email protected]>> wrote:
>> >
>> > Thanks Ahmet for calling out to the airflow folks. I believe,
I am able to follow their argument. So from my point of view I do
not have an issue with apache policy. But honestly still trying to
wrap my head around Roberts concern with rebuilding/resigning.
Currently, our actual release is only a tag on source repo and
promoting artefacts. Do not yet understand how that needs to change
to get PyPi included.
>>
>> It's not a big change, but let me clarify.
>>
>> Currently our release preparation goes something like this:
>>
>> 1) Check out the repo, update the versions to 2.x, build and
sign the artifacts.
>> 2) Announce these artifacts as rcN
>> 2a) Push the artifacts to SVN dev/...
>> 2b) Push artifacts to the apache maven repository.
>> 3) Depending on vote, go back to step (1) or forward to step (4).
>> 4) Copy these artifacts as the actual release.
>>
>> Now if we just try to add (2c) Push these artifacts to Pypi, it will
>> be treated (by pypi's tooling, anyone who downloads the tarball,
...)
>> as an actual release. You also can't re-push a tarball with the same
>> name and different contents (the idea being that named releases
should
>> never change). So we'd need to change step (1) to update the version
>> to 2.x.rcN *and* add a step in (4) to update the version to 2.x
(no rc
>> suffix), rebuild, resign before publishing.
>>
>> As mentioned, possibly the rcN suffix could be part of the building
>> step for Python.
>>
>> > On Wed, May 1, 2019 at 1:33 AM Ahmet Altay <[email protected]
<mailto:[email protected]>> wrote:
>> >>
>> >> Michael, Max and other folks who are concerned about the
compatibility with the apache release policy. Does the information
in this thread sufficiently address your concerns? Especially the
part where, the rc artifacts will be protected by a flag (i.e.
--pre) from general consumption.
>> >>
>> >> On Tue, Apr 30, 2019 at 3:59 PM Robert Bradshaw
<[email protected] <mailto:[email protected]>> wrote:
>> >>>
>> >>> On Tue, Apr 30, 2019 at 6:11 PM Ahmet Altay
<[email protected] <mailto:[email protected]>> wrote:
>> >>> >
>> >>> > This conversation get quite Python centric. Is there a
similar need for Java?
>> >>>
>> >>> I think Java is already covered. Go is a different story
(but the even
>> >>> versioning and releasing is being worked out).
>> >>>
>> >>> > On Tue, Apr 30, 2019 at 4:54 AM Robert Bradshaw
<[email protected] <mailto:[email protected]>> wrote:
>> >>> >>
>> >>> >> If we can, by the apache guidelines, post RCs to pypy that is
>> >>> >> definitely the way to go. (Note that test.pypi is for
developing
>> >>> >> against the pypi interface, not for pushing anything
real.) The caveat
>> >>> >> about naming these with rcN in the version number still
applies
>> >>> >> (that's how pypi guards them against non-explicit installs).
>> >>> >
>> >>> > Related to the caveat, I believe this can be easily
scripted or even made part of the travis/wheels pipeline to take the
release branch, edit the version string in place to add rc, and
build the necessary files.
>> >>>
>> >>> Yes. But the resulting artifacts would have to be rebuilt (and
>> >>> re-signed) without the version edit for the actual release.
(Well, we
>> >>> could possibly edit the artifacts rather than rebuild them.) And
>> >>> pushing un-edited ones early would be really bad. (It's the
classic
>> >>> tension of whether a pre-release should be marked internally or
>> >>> externally, re-publishing a new set of bits for the actual
release or
>> >>> re-using version numbers for different sets of bits. Pypi
does one,
>> >>> apache does another...)
>> >>>
>> >>> >> The advantage is that a user can do "pip install --pre
apache-beam" to
>> >>> >> get the latest rc rather than "pip install
>> >>> >>
https://dist.apache.org/repos/dist/dev/beam/changing/and/ephemeral/path";
>> >>> >>
>> >>> >> On Mon, Apr 29, 2019 at 11:34 PM Pablo Estrada
<[email protected] <mailto:[email protected]>> wrote:
>> >>> >> >
>> >>> >> > Aw that's interesting!
>> >>> >> >
>> >>> >> > I think, with these considerations, I am only
marginally more inclined towards publishing to test.pypi. That would
make me a +0.9 on publishing RCs to the main pip repo then.
>> >>> >> >
>> >>> >> > Thanks for doing the research Ahmet. :)
>> >>> >> > Best
>> >>> >> > -P
>> >>> >> >
>> >>> >> > On Mon, Apr 29, 2019 at 1:53 PM Ahmet Altay
<[email protected] <mailto:[email protected]>> wrote:
>> >>> >> >>
>> >>> >> >> I asked to Airflow folks about this. See [1] for the
full response and a link to one of their RC emails. To summarize
their position (specifically for pypi) is: Unless a user does
something explicit (such as using a flag, or explicitly requesting
an rc release), pip install will not serve RC binaries. And that is
compatible with RC section of
http://www.apache.org/legal/release-policy.html#release-types
>> >>> >> >>
>> >>> >> >> Ahmet
>> >>> >> >>
>> >>> >> >> [1]
https://lists.apache.org/thread.html/f1f342332c1e180f57d60285bebe614ffa77bb53c4f74c4cbc049096@%3Cdev.airflow.apache.org%3E
>> >>> >> >>
>> >>> >> >> On Fri, Apr 26, 2019 at 3:38 PM Ahmet Altay
<[email protected] <mailto:[email protected]>> wrote:
>> >>> >> >>>
>> >>> >> >>> The incremental value of publishing python artifacts
to a separate place but not to actual pypi listing will be low.
Users can already download RC artifacts, or even pip install from
http location directly. I think the incremental value will be low,
because for a user or a downstream library to test with Beam RCs
using their usual ways will still require them to get other
dependencies from the regular pypi listing. That would mean they
need to change their setup to test with beam rcs, which is the same
state as today. There will be some incremental value of putting them
in more obvious places (e.g. pypi test repository). I would rather
not complicate the release process for doing this.
>> >>> >> >>>
>> >>> >> >>>
>> >>> >> >>>
>> >>> >> >>> On Thu, Apr 25, 2019 at 2:25 PM Kenneth Knowles
<[email protected] <mailto:[email protected]>> wrote:
>> >>> >> >>>>
>> >>> >> >>>> Pip is also able to be pointed at any raw hosted
directory for the install, right? So we could publish RCs or
snapshots somewhere with more obvious caveats and not interfere with
the pypi list of actual releases. Much like the Java snapshots are
stored in a separate opt-in repository.
>> >>> >> >>>>
>> >>> >> >>>> Kenn
>> >>> >> >>>>
>> >>> >> >>>> On Thu, Apr 25, 2019 at 5:39 AM Maximilian Michels
<[email protected] <mailto:[email protected]>> wrote:
>> >>> >> >>>>>
>> >>> >> >>>>> > wouldn't that be in conflict with Apache release
policy [1] ?
>> >>> >> >>>>> > [1] http://www.apache.org/legal/release-policy.html
>> >>> >> >>>>>
>> >>> >> >>>>> Indeed, advertising pre-release artifacts is
against ASF rules. For
>> >>> >> >>>>> example, Flink was asked to remove a link to the
Maven snapshot
>> >>> >> >>>>> repository from their download page.
>> >>> >> >>>>>
>> >>> >> >>>>> However, that does not mean we cannot publish
Python artifacts. We just
>> >>> >> >>>>> have to clearly mark them for developers only and
not advertise them
>> >>> >> >>>>> alongside with the official releases.
>> >>> >> >>>>>
>> >>> >> >>>>> -Max
>> >>> >> >>>>>
>> >>> >> >>>>> On 25.04.19 10:23, Robert Bradshaw wrote:
>> >>> >> >>>>> > Don't we push java artifacts to maven
repositories as part of the RC
>> >>> >> >>>>> > process? And completely unvetted snapshots? (Or
is this OK because
>> >>> >> >>>>> > they are special opt-in apache-only ones?)
>> >>> >> >>>>> >
>> >>> >> >>>>> > I am generally in favor of the idea, but would
like to avoid increased
>> >>> >> >>>>> > toil on the release manager.
>> >>> >> >>>>> >
>> >>> >> >>>>> > One potential hitch I see is that current release
process updates the
>> >>> >> >>>>> > versions to x.y.z (no RC or other pre-release
indicator in the version
>> >>> >> >>>>> > number) whereas pypi (and other systems)
typically expect distinct
>> >>> >> >>>>> > (recognizable) version numbers for each attempt,
and only the actual
>> >>> >> >>>>> > final result has the actual final release version.
>> >>> >> >>>>> >
>> >>> >> >>>>> > On Thu, Apr 25, 2019 at 6:38 AM Ahmet Altay
<[email protected] <mailto:[email protected]>> wrote:
>> >>> >> >>>>> >>
>> >>> >> >>>>> >> I do not know the answer.I believe this will be
similar to sharing the RC artifacts for validation purposes and
would not be a formal release by itself. But I am not an expert and
I hope others will share their opinions.
>> >>> >> >>>>> >>
>> >>> >> >>>>> >> I quickly searched pypi for apache projects and
found at least airflow [1] and libcloud [2] are publishing rc
artifacts to pypi. We can reach out to those communities and learn
about their processes.
>> >>> >> >>>>> >>
>> >>> >> >>>>> >> Ahmet
>> >>> >> >>>>> >>
>> >>> >> >>>>> >> [1] https://pypi.org/project/apache-airflow/#history
>> >>> >> >>>>> >> [2]
https://pypi.org/project/apache-libcloud/#history
>> >>> >> >>>>> >>
>> >>> >> >>>>> >> On Wed, Apr 24, 2019 at 6:15 PM Michael Luckey
<[email protected] <mailto:[email protected]>> wrote:
>> >>> >> >>>>> >>>
>> >>> >> >>>>> >>> Hi,
>> >>> >> >>>>> >>>
>> >>> >> >>>>> >>> wouldn't that be in conflict with Apache
release policy [1] ?
>> >>> >> >>>>> >>>
>> >>> >> >>>>> >>> [1] http://www.apache.org/legal/release-policy.html
>> >>> >> >>>>> >>>
>> >>> >> >>>>> >>> On Thu, Apr 25, 2019 at 1:35 AM Alan Myrvold
<[email protected] <mailto:[email protected]>> wrote:
>> >>> >> >>>>> >>>>
>> >>> >> >>>>> >>>> Great idea. I like the RC candidates to follow
as much as the release artifact process as possible.
>> >>> >> >>>>> >>>>
>> >>> >> >>>>> >>>> On Wed, Apr 24, 2019 at 3:27 PM Ahmet Altay
<[email protected] <mailto:[email protected]>> wrote:
>> >>> >> >>>>> >>>>>
>> >>> >> >>>>> >>>>> To clarify my proposal, I am proposing
publishing to the production pypi repository with an rc tag in the
version. And in turn allow users to depend on beam's rc version +
all the other regular dependencies users would have directly from pypi.
>> >>> >> >>>>> >>>>>
>> >>> >> >>>>> >>>>> Publishing to test pypi repo would also be
helpful if test pypi repo also mirrors other packages that exist in
the production pypi repository.
>> >>> >> >>>>> >>>>>
>> >>> >> >>>>> >>>>> On Wed, Apr 24, 2019 at 3:12 PM Pablo Estrada
<[email protected] <mailto:[email protected]>> wrote:
>> >>> >> >>>>> >>>>>>
>> >>> >> >>>>> >>>>>> I think this is a great idea. A way of doing
it for python would be by using the test repository for PyPi[1], and
that way we would not have to do an official PyPi release, but still
would be able to install it with pip (by passing an extra flag), and
test.
>> >>> >> >>>>> >>>>>>
>> >>> >> >>>>> >>>>>> In fact, there are some Beam artifacts
already in there[2]. At some point I looked into this, but couldn't
figure out who has access/the password for it.
>> >>> >> >>>>> >>>>>
>> >>> >> >>>>> >>>>>
>> >>> >> >>>>> >>>>> I also don't know who owns beam package in
test pypi repo. Does anybody know?
>> >>> >> >>>>> >>>>>
>> >>> >> >>>>> >>>>>>
>> >>> >> >>>>> >>>>>>
>> >>> >> >>>>> >>>>>> In short: +1, and I would suggest using the
test PyPi repo to avoid publishing to the main PyPi repo.
>> >>> >> >>>>> >>>>>> Best
>> >>> >> >>>>> >>>>>> -P.
>> >>> >> >>>>> >>>>>>
>> >>> >> >>>>> >>>>>> [1] https://test.pypi.org/
>> >>> >> >>>>> >>>>>> [2] https://test.pypi.org/project/apache-beam/
>> >>> >> >>>>> >>>>>>
>> >>> >> >>>>> >>>>>> On Wed, Apr 24, 2019 at 3:04 PM Ahmet Altay
<[email protected] <mailto:[email protected]>> wrote:
>> >>> >> >>>>> >>>>>>>
>> >>> >> >>>>> >>>>>>> Hi all,
>> >>> >> >>>>> >>>>>>>
>> >>> >> >>>>> >>>>>>> What do you think about the idea of
publishing pre-release artifacts as part of the RC emails?
>> >>> >> >>>>> >>>>>>>
>> >>> >> >>>>> >>>>>>> For Python this would translate into
publishing the same artifacts from RC email with a version like
"2.X.0rcY" to pypi. I do not know, but I am guessing we can do a
similar thing with Maven central for Java artifacts as well.
>> >>> >> >>>>> >>>>>>>
>> >>> >> >>>>> >>>>>>> Advantages would be:
>> >>> >> >>>>> >>>>>>> - Allow end users to validate RCs for their
own purposes using the same exact process they will normally use.
>> >>> >> >>>>> >>>>>>> - Enable early-adaptors to start using RC
releases early on in the release cycle if that is what they would
like to do. This will in turn reduce time pressure on some releases.
Especially for cases like someone needs a release to be finalized
for an upcoming event.
>> >>> >> >>>>> >>>>>>>
>> >>> >> >>>>> >>>>>>> There will also be disadvantages, some I
could think of:
>> >>> >> >>>>> >>>>>>> - Users could request support for RC
artifacts. Hopefully in the form of feedback for us to improve the
release. But it could also be in the form of folks using RC
artifacts for production for a long time.
>> >>> >> >>>>> >>>>>>> - It will add toil to the current release
process, there will be one more step for each RC. I think for python
this will be a small step but nevertheless it will be additional work.
>> >>> >> >>>>> >>>>>>>
>> >>> >> >>>>> >>>>>>> For an example of this, you can take a look
at tensorflow releases. For 1.13 there were 3 pre-releases [1].
>> >>> >> >>>>> >>>>>>>
>> >>> >> >>>>> >>>>>>> Ahmet
>> >>> >> >>>>> >>>>>>>
>> >>> >> >>>>> >>>>>>> [1]
https://pypi.org/project/tensorflow/#history
