Re: Airflow voting on release artifacts

[Niclas Hedhman]

I think the problem is lying with "users cannot tell the difference"...
Possibly that the term "release candidate" is not aligned/defined along the
same semantics.

"Users", as in outside the committers, are NOT to download any
source/binary artifacts up for a vote. Before the vote artifacts are not to
be publicly available, such as on dist.apache.org or Python's global/public
repositories (which I blatantly assumes exist and pip works against those),
nor are committers expected to put these artifacts on local repositories
(also assuming those are possible).

"Committers", on the other hand are EXPECTED to understand what a release
candidate is, how it is treated and how it is discarded after the vote
(whether it passes or not, should not matter).

HTH
NIclas

On Tue, May 2, 2017 at 6:01 AM, Bolke de Bruin  wrote:

> Yes you can, but how do we verify it actually happened? Maven will, afaik,
> happily upgrade “apache-beam-1.0.0rc2” to “apache-beam-1.0.0” although they
> contain the exact same artefact. Pip won’t do that.
>
> If we use a release candidate named “apache-airflow-1.8.1rc2” while the
> package requires us to contain “apache-airflow-1.8.1” users cannot tell the
> difference if they installed RC2 or if it was the actual release. Worse
> even, we cannot tell the difference anymore. Then we just need to wait for
> the confusion in the bug reports.
>
> B.
>
> > On 1 May 2017, at 23:42, Stian Soiland-Reyes  wrote:
> >
> > Sorry for my ignorance, but is there no easy way with pip to uninstall
> the
> > package or force-install a new RC?
> >
> > If a previous RC failed the vote, then it should be uninstalled by
> everyone
> > anyway. In fact if you test a RC by installing it globally, then you
> should
> > always uninstall it after testing as you don't know the result of the
> vote
> > yet and should revert to the latest official release (or your own build
> > from scm).
> >
> > This is no different from Java/Maven - if you happen to test an RC by
> "mvn
> > install" (instead of "mvn verify") then you need to clean it out
> > afterwards. There is no easy command for it in mvn, although you can
> > usually just rm -rf the corresponding folder in .m2/repository.
> >
> >
> >
> > On 1 May 2017 10:00 pm, "Bolke de Bruin"  wrote:
> >
> >
> >> On 1 May 2017, at 22:39, Alex Harui  wrote:
> >>
> >>
> >>
> >> On 5/1/17, 11:44 AM, "Bolke de Bruin"  bdbr...@gmail.com>> wrote:
> >>
> >>>
>  On 1 May 2017, at 17:36, Alex Harui  wrote:
> 
> 
> 
>  On 5/1/17, 7:44 AM, "Hitesh Shah"  wrote:
> 
> > Hi Justin,
> >
> > Currently, the podling has been modifying the contents and hence this
> > discussion.
> 
>  I agree with Justin and others that modification after the vote is
> not a
>  good thing.  So my assumption was that if you add your 2a step and
>  modify
>  the binary before the vote, it will be acceptable.  IMO, all you need
>  is a
>  way to verify that the binary the voters test is essentially the same
> as
>  the binary you want to actually release.
> 
>  -Alex
> 
> 
> >>>
> >>> Hi Alex,
> >>>
> >>> As mentioned earlier this is not possible in a clean way. Version
> >>> information is contained within the source package and it is required
> by
> >>> specification to be. Installation happens from this source package.
> There
> >>> are no “binaries”.
> >>>
> >>> We understand the need to vote on the artefacts, however the way it is
> >>> required to work put us between a rock and a hard place: either our
> users
> >>> can end up with an outdated pre-release while reporting they have the
> >>> release installed or we need to vote 2+2 times (PMC+IPMC).
> >>>
> >>> We are looking to optimize this process either technically or
> >>> procedurally, but until so far haven’t been able to distill anything
> that
> >>> really helps.
> >>
> >> Well, I'm quite confused now.  Hitesh seems to say there are binaries.
> >> And I have proposed a couple of ideas where you create different
> artifacts
> >> for voters vs. customers that I think get around all of these issues.
> >> AFAIK, nobody on this list has objected to those proposals.
> >>
> >> Maybe there is something about Python I don't understand, but if I had
> to
> >> ship a set of Javascript files with an embedded version number in one of
> >> those files, I would use what I proposed.  AFAICT, there is no
> obligation
> >> to make your customers (not your voters) consume the source package, it
> >> just has to be possible to generate what the customers use from the
> source
> >> package.
> >>
> >
> > In Python we are used to install through so called source distributions
> > “sdist”. Package managers (e.g. pip) use the filename to determine
> whether
> > to download a new package and if they do they examine the contents of the
> > package to find out it 

Re: Airflow voting on release artifacts

[Alex Harui]

On 5/1/17, 2:00 PM, "Bolke de Bruin"  wrote:
>
>In Python we are used to install through so called source distributions
>“sdist”. Package managers (e.g. pip) use the filename to determine
>whether to download a new package and if they do they examine the
>contents of the package to find out it they need to install the package.
>They do this by examining the version contained inside the package. Thus
>while a different filename will trigger a new download, it might not
>install updated parts of the package. This is different from your example
>as no installer is examining both the name of the tar ball and the
>contents of your javascript files for a version identifier.
>
>But maybe you have a point. We could just do a "git clone”, update the
>version (not push it to git until final release), tar it. We then ask
>people to vote on it. Then we could provide the convenience package (that
>everyone will use) next to it. Or if we consider the “sdist” a binary
>release officially we vote on that one as well after the first vote. Two
>downsides to this are: if only option 1) nobody would user the tar, as
>the sdist is essentially the same and works with the package managers.
>Might be a bit excessive? 2) that would be a 2+2 vote again.
>
>Option 1 could work, it isn’t ideal, but will satisfy the procedure.

Many projects produce two packages: the actual sources, and something most
customers want to use.  The main reason ASF projects focus on the source
is because we are an "Open Source" foundation, but also because you can
verify that a source package isn't infected by a virus more easily than
reviewing other kinds of files.  AIUI, there are customers who are
concerned enough about the safety of a release that they only work with
source packages and compile everything from source.  So, whatever you call
your "source" package that is officially voted on must meet this criteria.

The customer package can be collection of files, but it must meet certain
requirements like having a LICENSE and NOTICE and detached signature that
the voters verify.

Many release examiners want to verify the source package against a Git
hash or SVN tag. That isn't a "must do", but a good thing if you can do
it.  So, I'm not sure delaying pushing the tag is a good idea.  What we do
in our project is pick whatever hash is the point in history for the
release and tag is as RC1, then when the vote passes, tag whatever hash
finally passes with the release tag.

So, if an sdist is entirely source, IMO, you can just make two sdist
packages (one for customers and one with "RC1" appended to the version
number) and offer both to the voters.  If the voters can easily validate
that the RC1 version they test is essentially the same as the other
package, then I think you are good to go.

HTH,
-Alex

Re: Airflow voting on release artifacts

[Stian Soiland-Reyes]

I understand that concern. Why do you think your users install release
candidates under vote? It might be time to think about splitting out a
users@ list?

Perhaps you can expand the instructions in the VOTE email with the exact
commands to install and uninstall, or a special test script that enforces
use of virtualenv for instance. With a larger community the VOTE emails
need to be more guiding for expectation management.

On 1 May 2017 11:01 pm, "Bolke de Bruin"  wrote:

> Yes you can, but how do we verify it actually happened? Maven will, afaik,
> happily upgrade “apache-beam-1.0.0rc2” to “apache-beam-1.0.0” although they
> contain the exact same artefact. Pip won’t do that.
>
> If we use a release candidate named “apache-airflow-1.8.1rc2” while the
> package requires us to contain “apache-airflow-1.8.1” users cannot tell the
> difference if they installed RC2 or if it was the actual release. Worse
> even, we cannot tell the difference anymore. Then we just need to wait for
> the confusion in the bug reports.
>
> B.
>
> > On 1 May 2017, at 23:42, Stian Soiland-Reyes  wrote:
> >
> > Sorry for my ignorance, but is there no easy way with pip to uninstall
> the
> > package or force-install a new RC?
> >
> > If a previous RC failed the vote, then it should be uninstalled by
> everyone
> > anyway. In fact if you test a RC by installing it globally, then you
> should
> > always uninstall it after testing as you don't know the result of the
> vote
> > yet and should revert to the latest official release (or your own build
> > from scm).
> >
> > This is no different from Java/Maven - if you happen to test an RC by
> "mvn
> > install" (instead of "mvn verify") then you need to clean it out
> > afterwards. There is no easy command for it in mvn, although you can
> > usually just rm -rf the corresponding folder in .m2/repository.
> >
> >
> >
> > On 1 May 2017 10:00 pm, "Bolke de Bruin"  wrote:
> >
> >
> >> On 1 May 2017, at 22:39, Alex Harui  wrote:
> >>
> >>
> >>
> >> On 5/1/17, 11:44 AM, "Bolke de Bruin"  bdbr...@gmail.com>> wrote:
> >>
> >>>
>  On 1 May 2017, at 17:36, Alex Harui  wrote:
> 
> 
> 
>  On 5/1/17, 7:44 AM, "Hitesh Shah"  wrote:
> 
> > Hi Justin,
> >
> > Currently, the podling has been modifying the contents and hence this
> > discussion.
> 
>  I agree with Justin and others that modification after the vote is
> not a
>  good thing.  So my assumption was that if you add your 2a step and
>  modify
>  the binary before the vote, it will be acceptable.  IMO, all you need
>  is a
>  way to verify that the binary the voters test is essentially the same
> as
>  the binary you want to actually release.
> 
>  -Alex
> 
> 
> >>>
> >>> Hi Alex,
> >>>
> >>> As mentioned earlier this is not possible in a clean way. Version
> >>> information is contained within the source package and it is required
> by
> >>> specification to be. Installation happens from this source package.
> There
> >>> are no “binaries”.
> >>>
> >>> We understand the need to vote on the artefacts, however the way it is
> >>> required to work put us between a rock and a hard place: either our
> users
> >>> can end up with an outdated pre-release while reporting they have the
> >>> release installed or we need to vote 2+2 times (PMC+IPMC).
> >>>
> >>> We are looking to optimize this process either technically or
> >>> procedurally, but until so far haven’t been able to distill anything
> that
> >>> really helps.
> >>
> >> Well, I'm quite confused now.  Hitesh seems to say there are binaries.
> >> And I have proposed a couple of ideas where you create different
> artifacts
> >> for voters vs. customers that I think get around all of these issues.
> >> AFAIK, nobody on this list has objected to those proposals.
> >>
> >> Maybe there is something about Python I don't understand, but if I had
> to
> >> ship a set of Javascript files with an embedded version number in one of
> >> those files, I would use what I proposed.  AFAICT, there is no
> obligation
> >> to make your customers (not your voters) consume the source package, it
> >> just has to be possible to generate what the customers use from the
> source
> >> package.
> >>
> >
> > In Python we are used to install through so called source distributions
> > “sdist”. Package managers (e.g. pip) use the filename to determine
> whether
> > to download a new package and if they do they examine the contents of the
> > package to find out it they need to install the package. They do this by
> > examining the version contained inside the package. Thus while a
> different
> > filename will trigger a new download, it might not install updated parts
> of
> > the package. This is different from your example as no installer is
> > examining both the name of the tar ball and the contents of 

Re: Airflow voting on release artifacts

[Bolke de Bruin]

Yes you can, but how do we verify it actually happened? Maven will, afaik, 
happily upgrade “apache-beam-1.0.0rc2” to “apache-beam-1.0.0” although they 
contain the exact same artefact. Pip won’t do that. 

If we use a release candidate named “apache-airflow-1.8.1rc2” while the package 
requires us to contain “apache-airflow-1.8.1” users cannot tell the difference 
if they installed RC2 or if it was the actual release. Worse even, we cannot 
tell the difference anymore. Then we just need to wait for the confusion in the 
bug reports.

B.

> On 1 May 2017, at 23:42, Stian Soiland-Reyes  wrote:
> 
> Sorry for my ignorance, but is there no easy way with pip to uninstall the
> package or force-install a new RC?
> 
> If a previous RC failed the vote, then it should be uninstalled by everyone
> anyway. In fact if you test a RC by installing it globally, then you should
> always uninstall it after testing as you don't know the result of the vote
> yet and should revert to the latest official release (or your own build
> from scm).
> 
> This is no different from Java/Maven - if you happen to test an RC by "mvn
> install" (instead of "mvn verify") then you need to clean it out
> afterwards. There is no easy command for it in mvn, although you can
> usually just rm -rf the corresponding folder in .m2/repository.
> 
> 
> 
> On 1 May 2017 10:00 pm, "Bolke de Bruin"  wrote:
> 
> 
>> On 1 May 2017, at 22:39, Alex Harui  wrote:
>> 
>> 
>> 
>> On 5/1/17, 11:44 AM, "Bolke de Bruin" > wrote:
>> 
>>> 
 On 1 May 2017, at 17:36, Alex Harui  wrote:
 
 
 
 On 5/1/17, 7:44 AM, "Hitesh Shah"  wrote:
 
> Hi Justin,
> 
> Currently, the podling has been modifying the contents and hence this
> discussion.
 
 I agree with Justin and others that modification after the vote is not a
 good thing.  So my assumption was that if you add your 2a step and
 modify
 the binary before the vote, it will be acceptable.  IMO, all you need
 is a
 way to verify that the binary the voters test is essentially the same as
 the binary you want to actually release.
 
 -Alex
 
 
>>> 
>>> Hi Alex,
>>> 
>>> As mentioned earlier this is not possible in a clean way. Version
>>> information is contained within the source package and it is required by
>>> specification to be. Installation happens from this source package. There
>>> are no “binaries”.
>>> 
>>> We understand the need to vote on the artefacts, however the way it is
>>> required to work put us between a rock and a hard place: either our users
>>> can end up with an outdated pre-release while reporting they have the
>>> release installed or we need to vote 2+2 times (PMC+IPMC).
>>> 
>>> We are looking to optimize this process either technically or
>>> procedurally, but until so far haven’t been able to distill anything that
>>> really helps.
>> 
>> Well, I'm quite confused now.  Hitesh seems to say there are binaries.
>> And I have proposed a couple of ideas where you create different artifacts
>> for voters vs. customers that I think get around all of these issues.
>> AFAIK, nobody on this list has objected to those proposals.
>> 
>> Maybe there is something about Python I don't understand, but if I had to
>> ship a set of Javascript files with an embedded version number in one of
>> those files, I would use what I proposed.  AFAICT, there is no obligation
>> to make your customers (not your voters) consume the source package, it
>> just has to be possible to generate what the customers use from the source
>> package.
>> 
> 
> In Python we are used to install through so called source distributions
> “sdist”. Package managers (e.g. pip) use the filename to determine whether
> to download a new package and if they do they examine the contents of the
> package to find out it they need to install the package. They do this by
> examining the version contained inside the package. Thus while a different
> filename will trigger a new download, it might not install updated parts of
> the package. This is different from your example as no installer is
> examining both the name of the tar ball and the contents of your javascript
> files for a version identifier.
> 
> But maybe you have a point. We could just do a "git clone”, update the
> version (not push it to git until final release), tar it. We then ask
> people to vote on it. Then we could provide the convenience package (that
> everyone will use) next to it. Or if we consider the “sdist” a binary
> release officially we vote on that one as well after the first vote. Two
> downsides to this are: if only option 1) nobody would user the tar, as the
> sdist is essentially the same and works with the package managers. Might be
> a bit excessive? 2) that would be a 2+2 vote again.
> 
> Option 1 could work, it isn’t ideal, but will satisfy the 

Re: Airflow voting on release artifacts

[Stian Soiland-Reyes]

Sorry for my ignorance, but is there no easy way with pip to uninstall the
package or force-install a new RC?

If a previous RC failed the vote, then it should be uninstalled by everyone
anyway. In fact if you test a RC by installing it globally, then you should
always uninstall it after testing as you don't know the result of the vote
yet and should revert to the latest official release (or your own build
from scm).

This is no different from Java/Maven - if you happen to test an RC by "mvn
install" (instead of "mvn verify") then you need to clean it out
afterwards. There is no easy command for it in mvn, although you can
usually just rm -rf the corresponding folder in .m2/repository.



On 1 May 2017 10:00 pm, "Bolke de Bruin"  wrote:


> On 1 May 2017, at 22:39, Alex Harui  wrote:
>
>
>
> On 5/1/17, 11:44 AM, "Bolke de Bruin"  wrote:
>
>>
>>> On 1 May 2017, at 17:36, Alex Harui  wrote:
>>>
>>>
>>>
>>> On 5/1/17, 7:44 AM, "Hitesh Shah"  wrote:
>>>
 Hi Justin,

 Currently, the podling has been modifying the contents and hence this
 discussion.
>>>
>>> I agree with Justin and others that modification after the vote is not a
>>> good thing.  So my assumption was that if you add your 2a step and
>>> modify
>>> the binary before the vote, it will be acceptable.  IMO, all you need
>>> is a
>>> way to verify that the binary the voters test is essentially the same as
>>> the binary you want to actually release.
>>>
>>> -Alex
>>>
>>>
>>
>> Hi Alex,
>>
>> As mentioned earlier this is not possible in a clean way. Version
>> information is contained within the source package and it is required by
>> specification to be. Installation happens from this source package. There
>> are no “binaries”.
>>
>> We understand the need to vote on the artefacts, however the way it is
>> required to work put us between a rock and a hard place: either our users
>> can end up with an outdated pre-release while reporting they have the
>> release installed or we need to vote 2+2 times (PMC+IPMC).
>>
>> We are looking to optimize this process either technically or
>> procedurally, but until so far haven’t been able to distill anything that
>> really helps.
>
> Well, I'm quite confused now.  Hitesh seems to say there are binaries.
> And I have proposed a couple of ideas where you create different artifacts
> for voters vs. customers that I think get around all of these issues.
> AFAIK, nobody on this list has objected to those proposals.
>
> Maybe there is something about Python I don't understand, but if I had to
> ship a set of Javascript files with an embedded version number in one of
> those files, I would use what I proposed.  AFAICT, there is no obligation
> to make your customers (not your voters) consume the source package, it
> just has to be possible to generate what the customers use from the source
> package.
>

In Python we are used to install through so called source distributions
“sdist”. Package managers (e.g. pip) use the filename to determine whether
to download a new package and if they do they examine the contents of the
package to find out it they need to install the package. They do this by
examining the version contained inside the package. Thus while a different
filename will trigger a new download, it might not install updated parts of
the package. This is different from your example as no installer is
examining both the name of the tar ball and the contents of your javascript
files for a version identifier.

But maybe you have a point. We could just do a "git clone”, update the
version (not push it to git until final release), tar it. We then ask
people to vote on it. Then we could provide the convenience package (that
everyone will use) next to it. Or if we consider the “sdist” a binary
release officially we vote on that one as well after the first vote. Two
downsides to this are: if only option 1) nobody would user the tar, as the
sdist is essentially the same and works with the package managers. Might be
a bit excessive? 2) that would be a 2+2 vote again.

Option 1 could work, it isn’t ideal, but will satisfy the procedure.

Bolke.

Re: Airflow voting on release artifacts

[Bolke de Bruin]

> On 1 May 2017, at 22:39, Alex Harui  wrote:
> 
> 
> 
> On 5/1/17, 11:44 AM, "Bolke de Bruin"  > wrote:
> 
>> 
>>> On 1 May 2017, at 17:36, Alex Harui  wrote:
>>> 
>>> 
>>> 
>>> On 5/1/17, 7:44 AM, "Hitesh Shah"  wrote:
>>> 
 Hi Justin,
 
 Currently, the podling has been modifying the contents and hence this
 discussion.
>>> 
>>> I agree with Justin and others that modification after the vote is not a
>>> good thing.  So my assumption was that if you add your 2a step and
>>> modify
>>> the binary before the vote, it will be acceptable.  IMO, all you need
>>> is a
>>> way to verify that the binary the voters test is essentially the same as
>>> the binary you want to actually release.
>>> 
>>> -Alex
>>> 
>>> 
>> 
>> Hi Alex,
>> 
>> As mentioned earlier this is not possible in a clean way. Version
>> information is contained within the source package and it is required by
>> specification to be. Installation happens from this source package. There
>> are no “binaries”.
>> 
>> We understand the need to vote on the artefacts, however the way it is
>> required to work put us between a rock and a hard place: either our users
>> can end up with an outdated pre-release while reporting they have the
>> release installed or we need to vote 2+2 times (PMC+IPMC).
>> 
>> We are looking to optimize this process either technically or
>> procedurally, but until so far haven’t been able to distill anything that
>> really helps.
> 
> Well, I'm quite confused now.  Hitesh seems to say there are binaries.
> And I have proposed a couple of ideas where you create different artifacts
> for voters vs. customers that I think get around all of these issues.
> AFAIK, nobody on this list has objected to those proposals.
> 
> Maybe there is something about Python I don't understand, but if I had to
> ship a set of Javascript files with an embedded version number in one of
> those files, I would use what I proposed.  AFAICT, there is no obligation
> to make your customers (not your voters) consume the source package, it
> just has to be possible to generate what the customers use from the source
> package.
> 

In Python we are used to install through so called source distributions 
“sdist”. Package managers (e.g. pip) use the filename to determine whether to 
download a new package and if they do they examine the contents of the package 
to find out it they need to install the package. They do this by examining the 
version contained inside the package. Thus while a different filename will 
trigger a new download, it might not install updated parts of the package. This 
is different from your example as no installer is examining both the name of 
the tar ball and the contents of your javascript files for a version 
identifier. 

But maybe you have a point. We could just do a "git clone”, update the version 
(not push it to git until final release), tar it. We then ask people to vote on 
it. Then we could provide the convenience package (that everyone will use) next 
to it. Or if we consider the “sdist” a binary release officially we vote on 
that one as well after the first vote. Two downsides to this are: if only 
option 1) nobody would user the tar, as the sdist is essentially the same and 
works with the package managers. Might be a bit excessive? 2) that would be a 
2+2 vote again.

Option 1 could work, it isn’t ideal, but will satisfy the procedure.

Bolke.

Re: Airflow voting on release artifacts

[Alex Harui]

On 5/1/17, 11:44 AM, "Bolke de Bruin"  wrote:

>
>> On 1 May 2017, at 17:36, Alex Harui  wrote:
>> 
>> 
>> 
>> On 5/1/17, 7:44 AM, "Hitesh Shah"  wrote:
>> 
>>> Hi Justin,
>>> 
>>> Currently, the podling has been modifying the contents and hence this
>>> discussion.
>> 
>> I agree with Justin and others that modification after the vote is not a
>> good thing.  So my assumption was that if you add your 2a step and
>>modify
>> the binary before the vote, it will be acceptable.  IMO, all you need
>>is a
>> way to verify that the binary the voters test is essentially the same as
>> the binary you want to actually release.
>> 
>> -Alex
>> 
>> 
>
>Hi Alex,
>
>As mentioned earlier this is not possible in a clean way. Version
>information is contained within the source package and it is required by
>specification to be. Installation happens from this source package. There
>are no “binaries”.
>
>We understand the need to vote on the artefacts, however the way it is
>required to work put us between a rock and a hard place: either our users
>can end up with an outdated pre-release while reporting they have the
>release installed or we need to vote 2+2 times (PMC+IPMC).
>
>We are looking to optimize this process either technically or
>procedurally, but until so far haven’t been able to distill anything that
>really helps.

Well, I'm quite confused now.  Hitesh seems to say there are binaries.
And I have proposed a couple of ideas where you create different artifacts
for voters vs. customers that I think get around all of these issues.
AFAIK, nobody on this list has objected to those proposals.

Maybe there is something about Python I don't understand, but if I had to
ship a set of Javascript files with an embedded version number in one of
those files, I would use what I proposed.  AFAICT, there is no obligation
to make your customers (not your voters) consume the source package, it
just has to be possible to generate what the customers use from the source
package.

HTH,
-Alex

Re: Airflow voting on release artifacts

[Bolke de Bruin]

> On 1 May 2017, at 17:36, Alex Harui  wrote:
> 
> 
> 
> On 5/1/17, 7:44 AM, "Hitesh Shah"  wrote:
> 
>> Hi Justin,
>> 
>> Currently, the podling has been modifying the contents and hence this
>> discussion.
> 
> I agree with Justin and others that modification after the vote is not a
> good thing.  So my assumption was that if you add your 2a step and modify
> the binary before the vote, it will be acceptable.  IMO, all you need is a
> way to verify that the binary the voters test is essentially the same as
> the binary you want to actually release.
> 
> -Alex
> 
> 

Hi Alex,

As mentioned earlier this is not possible in a clean way. Version information 
is contained within the source package and it is required by specification to 
be. Installation happens from this source package. There are no “binaries”.

We understand the need to vote on the artefacts, however the way it is required 
to work put us between a rock and a hard place: either our users can end up 
with an outdated pre-release while reporting they have the release installed or 
we need to vote 2+2 times (PMC+IPMC).

We are looking to optimize this process either technically or procedurally, but 
until so far haven’t been able to distill anything that really helps.

Cheers
Bolke



-
To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org
For additional commands, e-mail: general-h...@incubator.apache.org

Re: Airflow voting on release artifacts

[Alex Harui]

On 5/1/17, 7:44 AM, "Hitesh Shah"  wrote:

>Hi Justin,
>
>Currently, the podling has been modifying the contents and hence this
>discussion.

I agree with Justin and others that modification after the vote is not a
good thing.  So my assumption was that if you add your 2a step and modify
the binary before the vote, it will be acceptable.  IMO, all you need is a
way to verify that the binary the voters test is essentially the same as
the binary you want to actually release.

-Alex

Re: Airflow voting on release artifacts

[Hitesh Shah]

Hi Justin,

Currently, the podling has been modifying the contents and hence this
discussion.

thanks
-- Hitesh

On Thu, Apr 27, 2017 at 8:48 PM, Justin Mclean 
wrote:

> Hi,
>
> > The implication here is that the release manager is being trusted by the
> > PMC to release the modified convenience artifacts from the voted-upon
> > source without a new vote.
>
> How are the artefacts modified after the vote?
>
> IMO As long as the hashes and signature are the same there is no issue. If
> any of the contents change, rather than just files names, then it would no
> longer be a valid release.
>
> If it is just a rename, a safer way to rename the file this would be to do
> so when you do the svn move from dev staging area to the release area.
>
> Thanks,
> Justin
> -
> To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org
> For additional commands, e-mail: general-h...@incubator.apache.org
>
>

Re: Airflow voting on release artifacts

[Justin Mclean]

Hi,

> The implication here is that the release manager is being trusted by the
> PMC to release the modified convenience artifacts from the voted-upon
> source without a new vote.

How are the artefacts modified after the vote? 

IMO As long as the hashes and signature are the same there is no issue. If any 
of the contents change, rather than just files names, then it would no longer 
be a valid release. 

If it is just a rename, a safer way to rename the file this would be to do so 
when you do the svn move from dev staging area to the release area.

Thanks,
Justin
-
To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org
For additional commands, e-mail: general-h...@incubator.apache.org

Re: Airflow voting on release artifacts

[Alex Harui]

I'm not on the IPMC, so I don't have an official vote, but for me, it
would be ok if you include step 2a and the voters have a way to validate
that the 1.8.1rc1 binary they tested is "the same"=="identical except for
version numbers" as the 1.8.1 binaries you want to distribute to customers.

Also, I believe the source package build script must be able to reproduce
the 1.8.1 binary you want to distribute to customers.

So if you can do that I would hope the IPMC would approve.

-Alex

On 4/27/17, 3:57 PM, "Hitesh Shah"  wrote:

>Hi folks,
>
>Given that the source bits are the official release, would it be okay if
>the community as a whole decided on say the following approach:
>
>1) Bundle source with version set to 1.8.1
>2) Bundle binary convenience artifacts built using version set to 1.8.1rc1
>3) Publish source and binary bits to dist.a.o/dev for vote
>4) If vote passes, publish the source tarball that was voted upon and
>"modified" binary convenience artifacts built with version set to 1.8.1
>
>The implication here is that the release manager is being trusted by the
>PMC to release the modified convenience artifacts from the voted-upon
>source without a new vote.
>
>If it helps, there are a couple of variations of the above which could be
>applied to reduce the no. of total votes:
>
>2a) Create 2 binary sets for the vote - one with version 1.8.1rc1 and
>other
>with version 1.8.1 ( with only 1.8.1 being published on a successful vote)
>
>OR
>
>4a) After PPMC vote passes, use the original source and modified binaries
>for the IPMC vote and therefore the IPMC vote is on the final bits being
>published.
>
>Any comments?
>
>thanks
>-- Hitesh
>
>
>On Tue, Apr 25, 2017 at 9:02 PM, Alex Harui  wrote:
>
>>
>>
>> >
>> >3. There is no “separate” build script. Pip will just install a binary
>> >(“wheel”) or uses the source package (as shown above). Both are used
>> >interchangeable by users. We only distribute source packages at the
>> >moment.
>> >
>> >@Alex: I have to think a little bit more about what you wrote, but it
>>is
>> >currently confusing the hell out of me :). Furthermore, I am not sure
>>if
>> >it applies considering the above #3.
>>
>> It could still apply.  You would just have to add a build script that
>> renames the package and metadata.
>>
>> Let's say I wanted to release a single file that reported the version
>> number.  Forgive me that I don't know Python so I just grabbed what I
>> think it would look like from the internet.
>>
>> ---MyRelease.py---
>> print("I am version 1.2.3.")
>>
>> Let's assume this is what your "customers" want to use.
>>
>> I am proposing that the Apache Source Package also contain the following
>> file:
>>
>> ---BuildScript.sh---
>> # creates Customer Package in out folder.
>> mkdir out
>> sed s/1.2.3/1.2.3$1/ < MyRelease.py > out/MyRelease$1.py
>>
>> Voters would run:
>>
>>BuildScript.sh RC1
>>
>> That would result in:
>>
>> ---out/MyReleaseRC1.py---
>> print("I am version 1.2.3RC1.")
>>
>> And this version would be tested by the voters.  The source package
>>being
>> voted on contains the original MyRelease.py and BuildScript.sh.  The
>> release manager would also run:
>>
>>
>> BuildScript.sh
>>
>> That would result in:
>>
>> ---out/MyRelease.py---
>> print("I am version 1.2.3.")
>>
>> In our project, the RM posts the source package in the RC folder and
>> creates a folder called "binaries" for the compiled source.  You could
>> call the folder something else, but let's keep the names for now.  The
>>RM
>> would copy a zip of MyRelease.py and BuildScript.sh (and LICENSE,
>>NOTICE,
>> README) into the RC folder and out/MyRelease.py to the "binaries"
>>folder.
>> Along with signatures and checksum files.
>>
>>
>> Voters would download the zip, expand it, run "BuildScript.sh RC1" and
>> test with their out/MyReleaseRC1.py.  They would examine the zip to make
>> sure it is compliant with Apache release policy.  This is what all other
>> voters on all other projects generally do.  But they would perform one
>> different step, which is, instead of testing the MyRelease.py in the
>> "binaries" folder, they would simply diff their MyReleaseRC1.py against
>> the MyRelease.py in the "binaries" folder.  If the only diffs are the
>> version, they should feel satisfied that the resulting "customer"
>>packages
>> is ok for release.
>>
>> Of course, I could be wrong...
>>
>> HTH,
>> -Alex
>>
>>


-
To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org
For additional commands, e-mail: general-h...@incubator.apache.org

Re: Airflow voting on release artifacts

[Hitesh Shah]

Hi folks,

Given that the source bits are the official release, would it be okay if
the community as a whole decided on say the following approach:

1) Bundle source with version set to 1.8.1
2) Bundle binary convenience artifacts built using version set to 1.8.1rc1
3) Publish source and binary bits to dist.a.o/dev for vote
4) If vote passes, publish the source tarball that was voted upon and
"modified" binary convenience artifacts built with version set to 1.8.1

The implication here is that the release manager is being trusted by the
PMC to release the modified convenience artifacts from the voted-upon
source without a new vote.

If it helps, there are a couple of variations of the above which could be
applied to reduce the no. of total votes:

2a) Create 2 binary sets for the vote - one with version 1.8.1rc1 and other
with version 1.8.1 ( with only 1.8.1 being published on a successful vote)

OR

4a) After PPMC vote passes, use the original source and modified binaries
for the IPMC vote and therefore the IPMC vote is on the final bits being
published.

Any comments?

thanks
-- Hitesh


On Tue, Apr 25, 2017 at 9:02 PM, Alex Harui  wrote:

>
>
> >
> >3. There is no “separate” build script. Pip will just install a binary
> >(“wheel”) or uses the source package (as shown above). Both are used
> >interchangeable by users. We only distribute source packages at the
> >moment.
> >
> >@Alex: I have to think a little bit more about what you wrote, but it is
> >currently confusing the hell out of me :). Furthermore, I am not sure if
> >it applies considering the above #3.
>
> It could still apply.  You would just have to add a build script that
> renames the package and metadata.
>
> Let's say I wanted to release a single file that reported the version
> number.  Forgive me that I don't know Python so I just grabbed what I
> think it would look like from the internet.
>
> ---MyRelease.py---
> print("I am version 1.2.3.")
>
> Let's assume this is what your "customers" want to use.
>
> I am proposing that the Apache Source Package also contain the following
> file:
>
> ---BuildScript.sh---
> # creates Customer Package in out folder.
> mkdir out
> sed s/1.2.3/1.2.3$1/ < MyRelease.py > out/MyRelease$1.py
>
> Voters would run:
>
>BuildScript.sh RC1
>
> That would result in:
>
> ---out/MyReleaseRC1.py---
> print("I am version 1.2.3RC1.")
>
> And this version would be tested by the voters.  The source package being
> voted on contains the original MyRelease.py and BuildScript.sh.  The
> release manager would also run:
>
>
> BuildScript.sh
>
> That would result in:
>
> ---out/MyRelease.py---
> print("I am version 1.2.3.")
>
> In our project, the RM posts the source package in the RC folder and
> creates a folder called "binaries" for the compiled source.  You could
> call the folder something else, but let's keep the names for now.  The RM
> would copy a zip of MyRelease.py and BuildScript.sh (and LICENSE, NOTICE,
> README) into the RC folder and out/MyRelease.py to the "binaries" folder.
> Along with signatures and checksum files.
>
>
> Voters would download the zip, expand it, run "BuildScript.sh RC1" and
> test with their out/MyReleaseRC1.py.  They would examine the zip to make
> sure it is compliant with Apache release policy.  This is what all other
> voters on all other projects generally do.  But they would perform one
> different step, which is, instead of testing the MyRelease.py in the
> "binaries" folder, they would simply diff their MyReleaseRC1.py against
> the MyRelease.py in the "binaries" folder.  If the only diffs are the
> version, they should feel satisfied that the resulting "customer" packages
> is ok for release.
>
> Of course, I could be wrong...
>
> HTH,
> -Alex
>
>

Re: Airflow voting on release artifacts

[Alex Harui]

>
>3. There is no “separate” build script. Pip will just install a binary
>(“wheel”) or uses the source package (as shown above). Both are used
>interchangeable by users. We only distribute source packages at the
>moment.
>
>@Alex: I have to think a little bit more about what you wrote, but it is
>currently confusing the hell out of me :). Furthermore, I am not sure if
>it applies considering the above #3.

It could still apply.  You would just have to add a build script that
renames the package and metadata.

Let's say I wanted to release a single file that reported the version
number.  Forgive me that I don't know Python so I just grabbed what I
think it would look like from the internet.

---MyRelease.py---
print("I am version 1.2.3.")

Let's assume this is what your "customers" want to use.

I am proposing that the Apache Source Package also contain the following
file:

---BuildScript.sh---
# creates Customer Package in out folder.
mkdir out
sed s/1.2.3/1.2.3$1/ < MyRelease.py > out/MyRelease$1.py

Voters would run:

   BuildScript.sh RC1

That would result in:

---out/MyReleaseRC1.py---
print("I am version 1.2.3RC1.")

And this version would be tested by the voters.  The source package being
voted on contains the original MyRelease.py and BuildScript.sh.  The
release manager would also run:


BuildScript.sh

That would result in:

---out/MyRelease.py---
print("I am version 1.2.3.")

In our project, the RM posts the source package in the RC folder and
creates a folder called "binaries" for the compiled source.  You could
call the folder something else, but let's keep the names for now.  The RM
would copy a zip of MyRelease.py and BuildScript.sh (and LICENSE, NOTICE,
README) into the RC folder and out/MyRelease.py to the "binaries" folder.
Along with signatures and checksum files.


Voters would download the zip, expand it, run "BuildScript.sh RC1" and
test with their out/MyReleaseRC1.py.  They would examine the zip to make
sure it is compliant with Apache release policy.  This is what all other
voters on all other projects generally do.  But they would perform one
different step, which is, instead of testing the MyRelease.py in the
"binaries" folder, they would simply diff their MyReleaseRC1.py against
the MyRelease.py in the "binaries" folder.  If the only diffs are the
version, they should feel satisfied that the resulting "customer" packages
is ok for release.

Of course, I could be wrong...

HTH,
-Alex

Re: Airflow voting on release artifacts

[Jim Apple]

Would virtualenv help by avoiding the "Requirement already satisfied"
complaints?

On Tue, Apr 25, 2017 at 2:05 PM, Bolke de Bruin  wrote:

> Thank you thinking with us, that’s really appreciated. Unfortunately, many
> of the assumptions from the Java world do not apply to the Python world:
>
> Version information inside the artefact needs to be in sync with the
> external filename.
>
> Examples:
>
> 1. Plain rename of the tar ball
>
> bolke$ pip install airflow-1.8.0rc11+apache.incubating.tar.gz
> Processing ./airflow-1.8.0rc11+apache.incubating.tar.gz
>   Requirement already satisfied (use --upgrade to upgrade):
> airflow==1.8.0rc5+apache.incubating from file:///Users/bolke/Documents/
> dev/airflow/dist/airflow-1.8.0rc11%2Bapache.incubating.tar.gz in
> /Users/bolke/Documents/dev/airflow_env/lib/python2.7/site-packages
>
> Note that although RC11 is mentioned RC5 is contained in the package and
> is not upgraded.
>
> 2. Rename of the package inside the tar ball
> bolke$ mv airflow-1.8.0rc5+apache.incubating airflow-1.8.0rc11+apache.
> incubating
> bolke$ tar cfz ../airflow-1.8.0rc11+apache.incubating.tar.gz
> airflow-1.8.0rc11+apache.incubating/
>
> bolke$ pip install airflow-1.8.0rc11+apache.incubating.tar.gz
> Processing ./airflow-1.8.0rc11+apache.incubating.tar.gz
>   Requirement already satisfied (use --upgrade to upgrade):
> airflow==1.8.0rc5+apache.incubating from file:///Users/bolke/Documents/
> dev/airflow/dist/airflow-1.8.0rc11%2Bapache.incubating.tar.gz in
> /Users/bolke/Documents/dev/airflow_env/lib/python2.7/site-packages
>
> 3. There is no “separate” build script. Pip will just install a binary
> (“wheel”) or uses the source package (as shown above). Both are used
> interchangeable by users. We only distribute source packages at the moment.
>
> @Alex: I have to think a little bit more about what you wrote, but it is
> currently confusing the hell out of me :). Furthermore, I am not sure if it
> applies considering the above #3.
>
> At the moment we seem stuck between a rock and a hard place. And as we
> would like to release a new version pretty soon, we probably want to vote
> twice for now, and including the IPMC, four times. Hopefully we can have a
> vote in rapid succession :-).
>
> Bolke
>
>
> > On 25 Apr 2017, at 17:33, Alex Harui  wrote:
> >
> >
> >
> > On 4/25/17, 1:43 AM, "Bolke de Bruin" > wrote:
> >
> >> Hi Bertrand,
> >>
> >>> On 25 Apr 2017, at 09:04, Bertrand Delacretaz
> >>>  wrote:
> >>>
> >>> Hi,
> >>>
> >>> On Mon, Apr 24, 2017 at 10:18 PM, Chris Riccomini
> >>>  wrote:
>  ...Hitesh recently raised the issue that the artifact that passes the
>  vote
>  MUST be the one that we actually release...
> >>>
> >>> Yes in terms of having the same binary digests and signatures, but
> >>> renaming the files is fine IMO, especially for removing an -rc suffix
> >>> which makes total sense. I would just add that step to your release
> >>> process documentation to make it clear.
> >>>
>  ...Rename/rebuild after final vote (This is what Airflow is doing, and
>  Beam
>  does this, I believe)...
> >>>
> >>> I'd say rename yes but rebuild no, in order to keep the same digests
> >>> and signatures.
> >>>
> >>
> >> As mentioned earlier, that seems not to be possible. The metadata
> >> (filename) and version information inside the package need to be in
> sync.
> >> This how the python build tools and python ecosystem works.
> >
> > I'm not familiar with Python, but is it possible to have a command line
> > option that adds the "-rc" suffix in the right places?
> >
> > IMO, there are two "audiences".  One is the voters, and one is your
> > customers.  The customers should not be using the RC until after it is
> > approved unless they are volunteering to be a voter.  Voters are
> primarily
> > supposed to examine a source artifact, but if you also release a
> > "convenience binary" artifact, there are some examinations of that
> > "binary" artifact required.  For many projects this "convenience"
> artifact
> > is the one that the vast majority of customers consume.  AIUI, Python
> > doesn't have binaries, but IMO, there is no requirement that this
> > "convenience" artifact actually contains binaries.  A "convenience"
> > artifact is just supposed to be the source artifact processed by a build
> > script since many of your customer may not have, or may not want to
> > configure their machines to run whatever build script engine you choose
> > for your project.  Further, there is no requirement that I know of that
> > voters must test the "convenience" artifact by actually running it.  It
> > just makes sense to do so in most cases.  But if Python is interpreted
> > source, you may be able to use this to your advantage.
> >
> > So, your source package should consist of the source and build script
> > required to build this "customer"/"convenience" 

Re: Airflow voting on release artifacts

[Bolke de Bruin]

Thank you thinking with us, that’s really appreciated. Unfortunately, many of 
the assumptions from the Java world do not apply to the Python world:

Version information inside the artefact needs to be in sync with the external 
filename. 

Examples:

1. Plain rename of the tar ball

bolke$ pip install airflow-1.8.0rc11+apache.incubating.tar.gz
Processing ./airflow-1.8.0rc11+apache.incubating.tar.gz
  Requirement already satisfied (use --upgrade to upgrade): 
airflow==1.8.0rc5+apache.incubating from 
file:///Users/bolke/Documents/dev/airflow/dist/airflow-1.8.0rc11%2Bapache.incubating.tar.gz
 in /Users/bolke/Documents/dev/airflow_env/lib/python2.7/site-packages

Note that although RC11 is mentioned RC5 is contained in the package and is not 
upgraded.

2. Rename of the package inside the tar ball
bolke$ mv airflow-1.8.0rc5+apache.incubating airflow-1.8.0rc11+apache.incubating
bolke$ tar cfz ../airflow-1.8.0rc11+apache.incubating.tar.gz 
airflow-1.8.0rc11+apache.incubating/

bolke$ pip install airflow-1.8.0rc11+apache.incubating.tar.gz
Processing ./airflow-1.8.0rc11+apache.incubating.tar.gz
  Requirement already satisfied (use --upgrade to upgrade): 
airflow==1.8.0rc5+apache.incubating from 
file:///Users/bolke/Documents/dev/airflow/dist/airflow-1.8.0rc11%2Bapache.incubating.tar.gz
 in /Users/bolke/Documents/dev/airflow_env/lib/python2.7/site-packages

3. There is no “separate” build script. Pip will just install a binary 
(“wheel”) or uses the source package (as shown above). Both are used 
interchangeable by users. We only distribute source packages at the moment.

@Alex: I have to think a little bit more about what you wrote, but it is 
currently confusing the hell out of me :). Furthermore, I am not sure if it 
applies considering the above #3.

At the moment we seem stuck between a rock and a hard place. And as we would 
like to release a new version pretty soon, we probably want to vote twice for 
now, and including the IPMC, four times. Hopefully we can have a vote in rapid 
succession :-).

Bolke


> On 25 Apr 2017, at 17:33, Alex Harui  wrote:
> 
> 
> 
> On 4/25/17, 1:43 AM, "Bolke de Bruin"  > wrote:
> 
>> Hi Bertrand,
>> 
>>> On 25 Apr 2017, at 09:04, Bertrand Delacretaz
>>>  wrote:
>>> 
>>> Hi,
>>> 
>>> On Mon, Apr 24, 2017 at 10:18 PM, Chris Riccomini
>>>  wrote:
 ...Hitesh recently raised the issue that the artifact that passes the
 vote
 MUST be the one that we actually release...
>>> 
>>> Yes in terms of having the same binary digests and signatures, but
>>> renaming the files is fine IMO, especially for removing an -rc suffix
>>> which makes total sense. I would just add that step to your release
>>> process documentation to make it clear.
>>> 
 ...Rename/rebuild after final vote (This is what Airflow is doing, and
 Beam
 does this, I believe)...
>>> 
>>> I'd say rename yes but rebuild no, in order to keep the same digests
>>> and signatures.
>>> 
>> 
>> As mentioned earlier, that seems not to be possible. The metadata
>> (filename) and version information inside the package need to be in sync.
>> This how the python build tools and python ecosystem works.
> 
> I'm not familiar with Python, but is it possible to have a command line
> option that adds the "-rc" suffix in the right places?
> 
> IMO, there are two "audiences".  One is the voters, and one is your
> customers.  The customers should not be using the RC until after it is
> approved unless they are volunteering to be a voter.  Voters are primarily
> supposed to examine a source artifact, but if you also release a
> "convenience binary" artifact, there are some examinations of that
> "binary" artifact required.  For many projects this "convenience" artifact
> is the one that the vast majority of customers consume.  AIUI, Python
> doesn't have binaries, but IMO, there is no requirement that this
> "convenience" artifact actually contains binaries.  A "convenience"
> artifact is just supposed to be the source artifact processed by a build
> script since many of your customer may not have, or may not want to
> configure their machines to run whatever build script engine you choose
> for your project.  Further, there is no requirement that I know of that
> voters must test the "convenience" artifact by actually running it.  It
> just makes sense to do so in most cases.  But if Python is interpreted
> source, you may be able to use this to your advantage.
> 
> So, your source package should consist of the source and build script
> required to build this "customer"/"convenience" package, and the build
> script should allow a command line option to add the "-rc" suffix.  Then
> voters would be instructed to download the source package, and to test it,
> build a "customer" package with the "-rc" suffix and test the results.
> And voters would also be instructed to download the "customer" package
> that 

Re: Airflow voting on release artifacts

[Alex Harui]

On 4/25/17, 1:43 AM, "Bolke de Bruin"  wrote:

>Hi Bertrand,
>
>> On 25 Apr 2017, at 09:04, Bertrand Delacretaz
>> wrote:
>> 
>> Hi,
>> 
>> On Mon, Apr 24, 2017 at 10:18 PM, Chris Riccomini
>> wrote:
>>> ...Hitesh recently raised the issue that the artifact that passes the
>>>vote
>>> MUST be the one that we actually release...
>> 
>> Yes in terms of having the same binary digests and signatures, but
>> renaming the files is fine IMO, especially for removing an -rc suffix
>> which makes total sense. I would just add that step to your release
>> process documentation to make it clear.
>> 
>>> ...Rename/rebuild after final vote (This is what Airflow is doing, and
>>>Beam
>>> does this, I believe)...
>> 
>> I'd say rename yes but rebuild no, in order to keep the same digests
>> and signatures.
>> 
>
>As mentioned earlier, that seems not to be possible. The metadata
>(filename) and version information inside the package need to be in sync.
>This how the python build tools and python ecosystem works.

I'm not familiar with Python, but is it possible to have a command line
option that adds the "-rc" suffix in the right places?

IMO, there are two "audiences".  One is the voters, and one is your
customers.  The customers should not be using the RC until after it is
approved unless they are volunteering to be a voter.  Voters are primarily
supposed to examine a source artifact, but if you also release a
"convenience binary" artifact, there are some examinations of that
"binary" artifact required.  For many projects this "convenience" artifact
is the one that the vast majority of customers consume.  AIUI, Python
doesn't have binaries, but IMO, there is no requirement that this
"convenience" artifact actually contains binaries.  A "convenience"
artifact is just supposed to be the source artifact processed by a build
script since many of your customer may not have, or may not want to
configure their machines to run whatever build script engine you choose
for your project.  Further, there is no requirement that I know of that
voters must test the "convenience" artifact by actually running it.  It
just makes sense to do so in most cases.  But if Python is interpreted
source, you may be able to use this to your advantage.

So, your source package should consist of the source and build script
required to build this "customer"/"convenience" package, and the build
script should allow a command line option to add the "-rc" suffix.  Then
voters would be instructed to download the source package, and to test it,
build a "customer" package with the "-rc" suffix and test the results.
And voters would also be instructed to download the "customer" package
that doesn't have the -rc suffix.  But in order to test it, they only have
to diff that package against the "customer" package they built from the
source package.  It should be the same except for the metadata.

We'll see if others can find a problem with this plan, but IMO, that would
be good enough for me as an PMC voter.

HTH,
-Alex

Re: Airflow voting on release artifacts

[Niclas Hedhman]

It is Java, but we have version references internally in all kinds of
places.
So, what happens is that the build creates the "expected final release",
say "1.2.3" and sets all internal references to that. But the tarball will
be -RCx, which is voted upon. And then, as Bertrand suggested, a rename of
that tarball if the vote passes, is not a problem.

And as John points out, the release build tags "1.2.3" in git and we can
(re)move that later if the vote fails.


On Tue, Apr 25, 2017 at 5:18 PM, Bolke de Bruin  wrote:

> Hi Niclas,
>
> Is this Java or Python? I can only find Java for Polygene.
>
> Furthermore, how do you manage this you repository? Do you have the
> release already set in one of your files, e.g. something like this:
> https://github.com/apache/incubator-airflow/blob/v1-8-
> test/airflow/version.py  incubator-airflow/blob/v1-8-test/airflow/version.py>
>
> The build system generates the metadata from there, which is used by the
> package installers (e.g. pip).
>
> Cheers
> Bolke
>
>
> > On 25 Apr 2017, at 10:47, Niclas Hedhman  wrote:
> >
> > We have a similar issue in Polygene, but the internal version is simply
> the
> > expected version, say 1.2.3 and the RC has the different file name. No
> > packagers will ever get the -RC named artifact and no confusion is among
> > community members as they are aware of this. IF the RC passes, then the
> > rename can happen. IF the RC doesn't happen, you can rebuild with new
> > content and same internal version.
> >
> > Cheers
> > Niclas
> >
> > On Tue, Apr 25, 2017 at 4:43 PM, Bolke de Bruin 
> wrote:
> >
> >> Hi Bertrand,
> >>
> >>> On 25 Apr 2017, at 09:04, Bertrand Delacretaz <
> >> bdelacre...@codeconsult.ch> wrote:
> >>>
> >>> Hi,
> >>>
> >>> On Mon, Apr 24, 2017 at 10:18 PM, Chris Riccomini <
> criccom...@apache.org>
> >> wrote:
>  ...Hitesh recently raised the issue that the artifact that passes the
> >> vote
>  MUST be the one that we actually release...
> >>>
> >>> Yes in terms of having the same binary digests and signatures, but
> >>> renaming the files is fine IMO, especially for removing an -rc suffix
> >>> which makes total sense. I would just add that step to your release
> >>> process documentation to make it clear.
> >>>
>  ...Rename/rebuild after final vote (This is what Airflow is doing, and
> >> Beam
>  does this, I believe)...
> >>>
> >>> I'd say rename yes but rebuild no, in order to keep the same digests
> >>> and signatures.
> >>>
> >>
> >> As mentioned earlier, that seems not to be possible. The metadata
> >> (filename) and version information inside the package need to be in
> sync.
> >> This how the python build tools and python ecosystem works.
> >>
> >> - Bolke.
> >> -
> >> To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org
> >> For additional commands, e-mail: general-h...@incubator.apache.org
> >>
> >>
> >
> >
> > --
> > Niclas Hedhman, Software Developer
> > http://polygene.apache.org - New Energy for Java
>
>


-- 
Niclas Hedhman, Software Developer
http://polygene.apache.org - New Energy for Java

Re: Airflow voting on release artifacts

[John D. Ament]

I'll point out that this is a great conversation and set of questions to
have with your mentors first, the IPMC later if they didn't respond/give
you a clear path forward.  Did you engage with them?

I personally hate seeing rc's being put forward to the incubator.  Most of
us use git these days, branches are cheap.  Even in my $dayjob when rolling
a release we always create a release branch, cut the release from there.
If something broke delete it from our internal artifact repository and drop
the branch - effectively starting over.

Would following an approach like that work for you?

John

On Tue, Apr 25, 2017 at 5:18 AM Bolke de Bruin  wrote:

> Hi Niclas,
>
> Is this Java or Python? I can only find Java for Polygene.
>
> Furthermore, how do you manage this you repository? Do you have the
> release already set in one of your files, e.g. something like this:
>
> https://github.com/apache/incubator-airflow/blob/v1-8-test/airflow/version.py
> <
> https://github.com/apache/incubator-airflow/blob/v1-8-test/airflow/version.py
> >
>
> The build system generates the metadata from there, which is used by the
> package installers (e.g. pip).
>
> Cheers
> Bolke
>
>
> > On 25 Apr 2017, at 10:47, Niclas Hedhman  wrote:
> >
> > We have a similar issue in Polygene, but the internal version is simply
> the
> > expected version, say 1.2.3 and the RC has the different file name. No
> > packagers will ever get the -RC named artifact and no confusion is among
> > community members as they are aware of this. IF the RC passes, then the
> > rename can happen. IF the RC doesn't happen, you can rebuild with new
> > content and same internal version.
> >
> > Cheers
> > Niclas
> >
> > On Tue, Apr 25, 2017 at 4:43 PM, Bolke de Bruin 
> wrote:
> >
> >> Hi Bertrand,
> >>
> >>> On 25 Apr 2017, at 09:04, Bertrand Delacretaz <
> >> bdelacre...@codeconsult.ch> wrote:
> >>>
> >>> Hi,
> >>>
> >>> On Mon, Apr 24, 2017 at 10:18 PM, Chris Riccomini <
> criccom...@apache.org>
> >> wrote:
>  ...Hitesh recently raised the issue that the artifact that passes the
> >> vote
>  MUST be the one that we actually release...
> >>>
> >>> Yes in terms of having the same binary digests and signatures, but
> >>> renaming the files is fine IMO, especially for removing an -rc suffix
> >>> which makes total sense. I would just add that step to your release
> >>> process documentation to make it clear.
> >>>
>  ...Rename/rebuild after final vote (This is what Airflow is doing, and
> >> Beam
>  does this, I believe)...
> >>>
> >>> I'd say rename yes but rebuild no, in order to keep the same digests
> >>> and signatures.
> >>>
> >>
> >> As mentioned earlier, that seems not to be possible. The metadata
> >> (filename) and version information inside the package need to be in
> sync.
> >> This how the python build tools and python ecosystem works.
> >>
> >> - Bolke.
> >> -
> >> To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org
> >> For additional commands, e-mail: general-h...@incubator.apache.org
> >>
> >>
> >
> >
> > --
> > Niclas Hedhman, Software Developer
> > http://polygene.apache.org - New Energy for Java
>
>

Re: Airflow voting on release artifacts

[Bolke de Bruin]

Hi Niclas,

Is this Java or Python? I can only find Java for Polygene. 

Furthermore, how do you manage this you repository? Do you have the release 
already set in one of your files, e.g. something like this:
https://github.com/apache/incubator-airflow/blob/v1-8-test/airflow/version.py 


The build system generates the metadata from there, which is used by the 
package installers (e.g. pip).

Cheers
Bolke


> On 25 Apr 2017, at 10:47, Niclas Hedhman  wrote:
> 
> We have a similar issue in Polygene, but the internal version is simply the
> expected version, say 1.2.3 and the RC has the different file name. No
> packagers will ever get the -RC named artifact and no confusion is among
> community members as they are aware of this. IF the RC passes, then the
> rename can happen. IF the RC doesn't happen, you can rebuild with new
> content and same internal version.
> 
> Cheers
> Niclas
> 
> On Tue, Apr 25, 2017 at 4:43 PM, Bolke de Bruin  wrote:
> 
>> Hi Bertrand,
>> 
>>> On 25 Apr 2017, at 09:04, Bertrand Delacretaz <
>> bdelacre...@codeconsult.ch> wrote:
>>> 
>>> Hi,
>>> 
>>> On Mon, Apr 24, 2017 at 10:18 PM, Chris Riccomini 
>> wrote:
 ...Hitesh recently raised the issue that the artifact that passes the
>> vote
 MUST be the one that we actually release...
>>> 
>>> Yes in terms of having the same binary digests and signatures, but
>>> renaming the files is fine IMO, especially for removing an -rc suffix
>>> which makes total sense. I would just add that step to your release
>>> process documentation to make it clear.
>>> 
 ...Rename/rebuild after final vote (This is what Airflow is doing, and
>> Beam
 does this, I believe)...
>>> 
>>> I'd say rename yes but rebuild no, in order to keep the same digests
>>> and signatures.
>>> 
>> 
>> As mentioned earlier, that seems not to be possible. The metadata
>> (filename) and version information inside the package need to be in sync.
>> This how the python build tools and python ecosystem works.
>> 
>> - Bolke.
>> -
>> To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org
>> For additional commands, e-mail: general-h...@incubator.apache.org
>> 
>> 
> 
> 
> -- 
> Niclas Hedhman, Software Developer
> http://polygene.apache.org - New Energy for Java

Re: Airflow voting on release artifacts

[Niclas Hedhman]

We have a similar issue in Polygene, but the internal version is simply the
expected version, say 1.2.3 and the RC has the different file name. No
packagers will ever get the -RC named artifact and no confusion is among
community members as they are aware of this. IF the RC passes, then the
rename can happen. IF the RC doesn't happen, you can rebuild with new
content and same internal version.

Cheers
Niclas

On Tue, Apr 25, 2017 at 4:43 PM, Bolke de Bruin  wrote:

> Hi Bertrand,
>
> > On 25 Apr 2017, at 09:04, Bertrand Delacretaz <
> bdelacre...@codeconsult.ch> wrote:
> >
> > Hi,
> >
> > On Mon, Apr 24, 2017 at 10:18 PM, Chris Riccomini 
> wrote:
> >> ...Hitesh recently raised the issue that the artifact that passes the
> vote
> >> MUST be the one that we actually release...
> >
> > Yes in terms of having the same binary digests and signatures, but
> > renaming the files is fine IMO, especially for removing an -rc suffix
> > which makes total sense. I would just add that step to your release
> > process documentation to make it clear.
> >
> >> ...Rename/rebuild after final vote (This is what Airflow is doing, and
> Beam
> >> does this, I believe)...
> >
> > I'd say rename yes but rebuild no, in order to keep the same digests
> > and signatures.
> >
>
> As mentioned earlier, that seems not to be possible. The metadata
> (filename) and version information inside the package need to be in sync.
> This how the python build tools and python ecosystem works.
>
> - Bolke.
> -
> To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org
> For additional commands, e-mail: general-h...@incubator.apache.org
>
>


-- 
Niclas Hedhman, Software Developer
http://polygene.apache.org - New Energy for Java

Re: Airflow voting on release artifacts

[Bolke de Bruin]

Hi Bertrand,

> On 25 Apr 2017, at 09:04, Bertrand Delacretaz  
> wrote:
> 
> Hi,
> 
> On Mon, Apr 24, 2017 at 10:18 PM, Chris Riccomini  
> wrote:
>> ...Hitesh recently raised the issue that the artifact that passes the vote
>> MUST be the one that we actually release...
> 
> Yes in terms of having the same binary digests and signatures, but
> renaming the files is fine IMO, especially for removing an -rc suffix
> which makes total sense. I would just add that step to your release
> process documentation to make it clear.
> 
>> ...Rename/rebuild after final vote (This is what Airflow is doing, and Beam
>> does this, I believe)...
> 
> I'd say rename yes but rebuild no, in order to keep the same digests
> and signatures.
> 

As mentioned earlier, that seems not to be possible. The metadata (filename) 
and version information inside the package need to be in sync. This how the 
python build tools and python ecosystem works.

- Bolke.
-
To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org
For additional commands, e-mail: general-h...@incubator.apache.org

Re: Airflow voting on release artifacts

[Bertrand Delacretaz]

Hi,

On Mon, Apr 24, 2017 at 10:18 PM, Chris Riccomini  wrote:
> ...Hitesh recently raised the issue that the artifact that passes the vote
> MUST be the one that we actually release...

Yes in terms of having the same binary digests and signatures, but
renaming the files is fine IMO, especially for removing an -rc suffix
which makes total sense. I would just add that step to your release
process documentation to make it clear.

> ...Rename/rebuild after final vote (This is what Airflow is doing, and Beam
> does this, I believe)...

I'd say rename yes but rebuild no, in order to keep the same digests
and signatures.

-Bertrand

-
To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org
For additional commands, e-mail: general-h...@incubator.apache.org

Re: Airflow voting on release artifacts

[Bolke de Bruin]

Hi John,

Typically one handles a “release candidate” equal to a release, except for the 
fact that it hasn’t been fully tested and reviewed yet. In case everyone (all 
entities) is satisfied with the RC it is then just rebranded to Release. 
Unfortunately, in the python ecosystem the version information is contained 
within the (source) package and as metadata by means of the filename. These two 
need to be in sync, otherwise package installers will get confused.

So imagine, if we started voting on “apache-airflow-1.8.1+incubating”, but it 
does *not* pass the vote due to a blocker bug or a license issue. We would need 
to restart the vote with an updated package but with the same name. 
Semantically, this is already confusing as you would need to examine the 
signatures to see if the version has changed. It is also risky for the 
users/testers and therefore for the developers: the new version will not 
trigger an upgrade with package installers and a new installation would need to 
be forced (contrary to how maven handles things for example). This means, we 
could receive bug reports on “1.8.1” which actually are “1.8.1-RC1” or 
“1.8.1-RC2” etc. 

In Apache we vote on artefacts. A rebrand delivers us two different artefacts: 
RC + Release. For us, as a podling, that would mean 4 votes: 2 at the PMC and 2 
at the IPMC. This is very time consuming for a very minor change.

Anecdotical evidence from two other projects shows that Apache Beam voted on 
0.6.0-RC2 as a release and then rebranded and updated packages after the vote 
(please correct me if I am wrong, just did a mailinglist scan and a comparison 
on the trees). Apache Libcloud (a pure python project) votes twice, but they 
seem to handle RCs differently, as there are large time gaps between the votes.

So we are seeking advice how to make efficient use of the Apache process, given 
the limitations of the ecosystem. I prefer what Apache Beam seems to be doing. 
I can also imagine to combine two artefacts in one vote: pointing everyone to 
the RC including it as a tar ball, but also including the tar ball and 
signatures for the *potential* rebranded version to release. Somehow we would 
need to hide the *potential* release as much from plain sight as possible, as 
we would have no way to distinguish between the actual release and the 
potential.

Bolke 

> On 25 Apr 2017, at 02:11, John D. Ament  wrote:
> 
> Do you need to cut a vote with something named rc?  Why can't you just use
> the version #?
> 
> On Mon, Apr 24, 2017 at 4:18 PM Chris Riccomini 
> wrote:
> 
>> Hey all,
>> 
>> We've had a question arise in the Airflow project. We're currently cutting
>> release candidates (RCs), and voting on those. These RCs contain an
>> artifact with the suffix -rc0, -rc1, etc. The problem with this is that the
>> final RC that passes the vote still has an -rcX in its version number. To
>> get around this, the release manager has just been rebuilding the .asc,
>> .sha, etc, and publishing this new artifact as the final release.
>> 
>> Hitesh recently raised the issue that the artifact that passes the vote
>> MUST be the one that we actually release. The problem with this is that it
>> would force us to publish multiple RCs with the exact same version number.
>> This is a terrible experience from a developer and user point of view.
>> Airflow is a Python project, and many users pip install various versions of
>> Airflow to test things out. Having multiple RCs with the same version
>> number is going to make it really difficult to keep things straight, both
>> during the VOTE, and afterwards.
>> 
>> Based on a cursory look, it seems that other projects handle this problem
>> in one of three ways:
>> 
>> * Double vote (the final RC gets voted AGAIN, this time, with proper
>> naming)
>> * Rename/rebuild after final vote (This is what Airflow is doing, and Beam
>> does this, I believe)
>> * All RCs have the exact same version number
>> 
>> Most Java projects are just publishing multiple RCs with the exact same
>> version number. This seems bad from a general software engineering
>> perspective, since it's going to cause build cache issues (e.g. .mvn
>> already contains version 1.2.3, even though a new RC with the same version
>> should be installed). Beam does seem to be using -SNAPSHOT releases in Java
>> for RCs, which is CLOSER to what we've been doing in Airflow (-rc0, -rc1,
>> etc), and they seem to be renaming after the release, just like we were.
>> 
>> What's the guidance here? We seem to have two requirements that are at odds
>> with eachtother:
>> 
>> 1) The RC artifact that we vote on is the one that we release
>> 2) We have real version numbers for RCs which are independent of one
>> another so build systems, dependency managements systems, and package
>> management systems can properly handle them.
>> 
>> Any help or thoughts would be appreciated.
>> 
>> Cheers,
>> Chris
>> 

Re: Airflow voting on release artifacts

[John D. Ament]

Do you need to cut a vote with something named rc?  Why can't you just use
the version #?

On Mon, Apr 24, 2017 at 4:18 PM Chris Riccomini 
wrote:

> Hey all,
>
> We've had a question arise in the Airflow project. We're currently cutting
> release candidates (RCs), and voting on those. These RCs contain an
> artifact with the suffix -rc0, -rc1, etc. The problem with this is that the
> final RC that passes the vote still has an -rcX in its version number. To
> get around this, the release manager has just been rebuilding the .asc,
> .sha, etc, and publishing this new artifact as the final release.
>
> Hitesh recently raised the issue that the artifact that passes the vote
> MUST be the one that we actually release. The problem with this is that it
> would force us to publish multiple RCs with the exact same version number.
> This is a terrible experience from a developer and user point of view.
> Airflow is a Python project, and many users pip install various versions of
> Airflow to test things out. Having multiple RCs with the same version
> number is going to make it really difficult to keep things straight, both
> during the VOTE, and afterwards.
>
> Based on a cursory look, it seems that other projects handle this problem
> in one of three ways:
>
> * Double vote (the final RC gets voted AGAIN, this time, with proper
> naming)
> * Rename/rebuild after final vote (This is what Airflow is doing, and Beam
> does this, I believe)
> * All RCs have the exact same version number
>
> Most Java projects are just publishing multiple RCs with the exact same
> version number. This seems bad from a general software engineering
> perspective, since it's going to cause build cache issues (e.g. .mvn
> already contains version 1.2.3, even though a new RC with the same version
> should be installed). Beam does seem to be using -SNAPSHOT releases in Java
> for RCs, which is CLOSER to what we've been doing in Airflow (-rc0, -rc1,
> etc), and they seem to be renaming after the release, just like we were.
>
> What's the guidance here? We seem to have two requirements that are at odds
> with eachtother:
>
> 1) The RC artifact that we vote on is the one that we release
> 2) We have real version numbers for RCs which are independent of one
> another so build systems, dependency managements systems, and package
> management systems can properly handle them.
>
> Any help or thoughts would be appreciated.
>
> Cheers,
> Chris
>