Re: Proposal to add "release requirements" check

[Jarek Potiuk]

It's an interesting (but not practical) idea to base the set of active
branches on the links in documentation pages.I would never try to do
that really and it was never anywhere close to my proposal - scanning
download pages is only to check if they have proper links/closer.lua
script, nothing else.

I think I am not looking for a perfect solution from day one which
will solve all cases. Problem with that is the same as with DAOs -
assuming that we have them is nice, but practically that does not work
as it requires upfront work from the projects. This makes them
essentially useless because projects do not have them and they are not
updated. It's a bit of a chicken-egg problem for all systems that
require every independent "project" in the organisation to implement
something in order to see the benefits of it. You cannot really make
use of a DAO to get "common view" of the projects until you are sure
that a) they are there b) they are updated regularly and that requires
the interested "projects" keep them updated as part of their
processes. I have no doubt implementing a new common process across
all the projects of ASF is next to impossible if you want to add a
"new" process to follow.  I am not "that" stupid to not realize that.

My idea is different - to tap as much as possible into existing
processes and make it a one-off effort for the project to comply. The
idea is that If it is flagged at board review time (automatically)
that the project does not follow rules, they will get notified and
after they correct their existing process/configuration/process it
will continue to work without change (and they will see if they are
compliant).

Tapping into existing processes:

1) updating URL to release page at "release submission page" seems
doable (and there are various ways you can make it easy and natural)
2) "downloads.apache.org" and "archives.apache.org" are already
updated (same process) - detecting artifacts that should be removed
from downloads - seems quite possible. They can look differently for
different projects. But my assumption is (something to work out over
time) that there are just a few patterns of versions to follow.

So coming back to point 1)  we might ask the projects to (again at
release submission time) choose which pattern they use (Semver or
other) - and even more explicitly ask them if there are releases that
should be made obsolete/inactive. That will require a bit of guessing
and mistakes initially when we add more projects, but - again - there
is a board review process that we can tap into. Every time there is a
board review time, we can look at the projects that are coming and add
a pattern or two if missing - and during the first review to ask them
about old releases. Eventually after a full round, the only effort
will be to add new projects that graduate, but then they will have
enough patterns to choose from at the first release

This means that we can have an "eventually consistent" system after
sufficient time passes - the important thing is that we can do it
gradually and over time. Even if we add a few projects a month,
eventually we will get there.

BTW. I actually tried to submit DAO for Airflow and failed miserably
(it was a long time ago and I don't remember details but tooling
set-up was not worth it and I gave up finally - also because I could
not see what DAO submission brings us or the ASF (precisely the
chicken-egg problem)

On Mon, Nov 8, 2021 at 10:54 PM sebb  wrote:
>
> On Mon, 8 Nov 2021 at 17:49, Jarek Potiuk  wrote:
> >
> > > However this does not solve the issue of stale download pages: how do
> > you know when a release version is no longer supported?
> >
> > What do you mean?
> >
> > Is there a requirement for that that I missed? I believe the
> > requirement is only about removing artifacts from "downloads.a.o" and
> > this is a completely different issue.
>
> That *is* what I am referring to.
> In order to remove artifacts from downloads.a.o, it needs to be
> established whether or not the release is still supported.
>
> If a release does not appear in any download page, then it is a
> candidate for removal (unless it is brand new).
>
> However if it does appear as a link to downloads.a.o in a download
> page, that does not necessarily mean that it is still a supported
> release.
>
> This is because projects don't always keep their download pages up to
> date, especially where there are individual pages for each release.
>
> For example, Airflow has download pages for 2.2.1 and 2.2.0; these
> both link to files on downloads.a.o.
>
> I suppose it is possible that there are plans to support updates to
> 2.2.0 separately from 2.2.1, but that seems unlikely, assuming that
> the project uses semantic versioning or something similar.
>
> > On Mon, Nov 8, 2021 at 6:42 PM sebb  wrote:
> > >
> > > On Mon, 8 Nov 2021 at 17:11, Jarek Potiuk  wrote:
> > > >
> > > > Yeah. We will have different URLs especially as more artifacts are
> > > > produced (airflow has 3 

Re: Proposal to add "release requirements" check

[sebb]

On Mon, 8 Nov 2021 at 17:49, Jarek Potiuk  wrote:
>
> > However this does not solve the issue of stale download pages: how do
> you know when a release version is no longer supported?
>
> What do you mean?
>
> Is there a requirement for that that I missed? I believe the
> requirement is only about removing artifacts from "downloads.a.o" and
> this is a completely different issue.

That *is* what I am referring to.
In order to remove artifacts from downloads.a.o, it needs to be
established whether or not the release is still supported.

If a release does not appear in any download page, then it is a
candidate for removal (unless it is brand new).

However if it does appear as a link to downloads.a.o in a download
page, that does not necessarily mean that it is still a supported
release.

This is because projects don't always keep their download pages up to
date, especially where there are individual pages for each release.

For example, Airflow has download pages for 2.2.1 and 2.2.0; these
both link to files on downloads.a.o.

I suppose it is possible that there are plans to support updates to
2.2.0 separately from 2.2.1, but that seems unlikely, assuming that
the project uses semantic versioning or something similar.

> On Mon, Nov 8, 2021 at 6:42 PM sebb  wrote:
> >
> > On Mon, 8 Nov 2021 at 17:11, Jarek Potiuk  wrote:
> > >
> > > Yeah. We will have different URLs especially as more artifacts are
> > > produced (airflow has 3 of those each with different download pages).
> > >
> > > But this should be straightforward when you can add the url when
> > > registering the release (as Sebb proposed). According to the release
> > > process you should register each such "release type" separately and
> > > adding a URL there will do the job nicely. And initially it can be
> > > optional and once we figure out how to communicate it best and maybe
> > > try with some projects first we can make it obligatory at release
> > > registration. We could make it convenient - for example an easy way to
> > > copy a previously added URL from selected previous release or
> > > autocomplete from previous entries - making the process of entry as
> > > smooth (or even smoother) than it is today.
> >
> > Or you could just ask for the URL if the release is not already on a
> > download page.
> > This would mean parsing the page of course, but could be used to
> > provide immediate feedback if the page is non-conforming.
> >
> > Asking for the page might also nudge projects to move to generic pages
> > rather than one per release.
> >
> > However this does not solve the issue of stale download pages: how do
> > you know when a release version is no longer supported?
> >
> > > J.
> > >
> > >
> > > On Mon, Nov 8, 2021 at 4:19 PM sebb  wrote:
> > > >
> > > > On Mon, 8 Nov 2021 at 11:29, Hans Van Akelyen
> > > >  wrote:
> > > > >
> > > > > Hi,
> > > > >
> > > > > I think a certain level of uniformity would not be bad for this.
> > > > > Having a fixed path to the downloads for each project makes it 
> > > > > simpler for
> > > > > the users to "shop" for Apache projects.
> > > > >
> > > > > Having .a.o/download/ as a link that should point to the 
> > > > > resources
> > > > > makes it simpler for everyone.
> > > >
> > > > I agree, but expect a lot of complaints if you try to get this enforced.
> > > > (and there's not much point unless it is enforced).
> > > >
> > > > > Kind Regards,
> > > > > Hans
> > > > >
> > > > > On Mon, 8 Nov 2021 at 12:19, Justin Mclean  
> > > > > wrote:
> > > > >
> > > > > > Hi,
> > > > > >
> > > > > > > However that assumes that you know where to find the download 
> > > > > > > page(s).
> > > > > > > It also assumes that projects update their download page(s) to 
> > > > > > > only
> > > > > > > show current releases.
> > > > > >
> > > > > > It does indeed assume that, and a few other things. Currently policy
> > > > > > doesn't specify what the download page URL should be, but once 
> > > > > > known I
> > > > > > would guess it wouldn’t change often.
> > > > > >
> > > > > > Kind Regards,
> > > > > > Justin
> > > > > >
> > > > > >
> > > > > >
> > > > > > -
> > > > > > To unsubscribe, e-mail: dev-unsubscr...@community.apache.org
> > > > > > For additional commands, e-mail: dev-h...@community.apache.org
> > > > > >
> > > > > >
> > > >
> > > > -
> > > > To unsubscribe, e-mail: dev-unsubscr...@community.apache.org
> > > > For additional commands, e-mail: dev-h...@community.apache.org
> > > >
> > >
> > > -
> > > To unsubscribe, e-mail: dev-unsubscr...@community.apache.org
> > > For additional commands, e-mail: dev-h...@community.apache.org
> > >
> >
> > -
> > To unsubscribe, e-mail: dev-unsubscr...@community.apache.org
> > For additional comman

Re: Proposal to add "release requirements" check

[Hans Van Akelyen]

I was also talking more from a user perspective.
It's not that hard to create a one pager that contains the latest version,
release dates, release notes and keys for all packages the project provides.

In the case of Airflow it's pretty much what you put in announcements, and
this one is fairly obvious to find.
On some other projects you really have to start digging.

And I am not sure I would want yet another xml/json/yaml config file that
has to be maintained when creating a release.
eg. you should update your DOAP file too (
https://projects.apache.org/project.html?airflow) as Dave said, many
projects fail to do so.

Maybe the download page url should be configurable (use the DOAP?), but I
would rather have a uniform way of finding that page.

On Mon, 8 Nov 2021 at 19:11, Dave Fisher  wrote:

>
>
> > On Nov 8, 2021, at 9:48 AM, Jarek Potiuk  wrote:
> >
> >> However this does not solve the issue of stale download pages: how do
> > you know when a release version is no longer supported?
> >
> > What do you mean?
> >
> > Is there a requirement for that that I missed? I believe the
> > requirement is only about removing artifacts from "downloads.a.o" and
> > this is a completely different issue.
>
> There is no definitive rule about the path to the download page. There is
> no standard and with over 200 projects, 35 podlings, and lots of products
> change is difficult.
>
> There is one definite requirement that can be checked. Whether the
> releases in svn are properly formed.
>
> We do this in the Incubator with the Clutch Analysis.
> https://svn.apache.org/repos/asf/incubator/public/trunk/clutch2.py - L1132
> Output found by clicking on a podling on
> https://incubator.apache.org/clutch/
>
> We also have code in ASF Pelican -
> https://github.com/apache/infrastructure-pelican/blob/master/plugins/asfdata.py#L362
> That was used to produce this example for Hadoop -
> https://template.staged.apache.org/downloads.html
> Using this template -
> https://github.com/apache/template-site/blob/main/content/downloads.ezmd
> Control is -
> https://github.com/apache/template-site/blob/main/asfdata.yaml#L110
>
> To review projects you might find out where their download page is by
> reviewing the project DOAP file, but we’ve never been able to get all
> projects to maintain one.
>
> Maybe it’s time to more carefully define what the ask is here.
>
> >
> > On Mon, Nov 8, 2021 at 6:42 PM sebb  wrote:
> >>
> >> On Mon, 8 Nov 2021 at 17:11, Jarek Potiuk  wrote:
> >>>
> >>> Yeah. We will have different URLs especially as more artifacts are
> >>> produced (airflow has 3 of those each with different download pages).
> >>>
> >>> But this should be straightforward when you can add the url when
> >>> registering the release (as Sebb proposed). According to the release
> >>> process you should register each such "release type" separately and
> >>> adding a URL there will do the job nicely. And initially it can be
> >>> optional and once we figure out how to communicate it best and maybe
> >>> try with some projects first we can make it obligatory at release
> >>> registration. We could make it convenient - for example an easy way to
> >>> copy a previously added URL from selected previous release or
> >>> autocomplete from previous entries - making the process of entry as
> >>> smooth (or even smoother) than it is today.
> >>
> >> Or you could just ask for the URL if the release is not already on a
> >> download page.
> >> This would mean parsing the page of course, but could be used to
> >> provide immediate feedback if the page is non-conforming.
> >>
> >> Asking for the page might also nudge projects to move to generic pages
> >> rather than one per release.
> >>
> >> However this does not solve the issue of stale download pages: how do
> >> you know when a release version is no longer supported?
> >>
> >>> J.
> >>>
> >>>
> >>> On Mon, Nov 8, 2021 at 4:19 PM sebb  wrote:
> 
>  On Mon, 8 Nov 2021 at 11:29, Hans Van Akelyen
>   wrote:
> >
> > Hi,
> >
> > I think a certain level of uniformity would not be bad for this.
> > Having a fixed path to the downloads for each project makes it
> simpler for
> > the users to "shop" for Apache projects.
> >
> > Having .a.o/download/ as a link that should point to the
> resources
> > makes it simpler for everyone.
> 
>  I agree, but expect a lot of complaints if you try to get this
> enforced.
>  (and there's not much point unless it is enforced).
> 
> > Kind Regards,
> > Hans
> >
> > On Mon, 8 Nov 2021 at 12:19, Justin Mclean 
> wrote:
> >
> >> Hi,
> >>
> >>> However that assumes that you know where to find the download
> page(s).
> >>> It also assumes that projects update their download page(s) to only
> >>> show current releases.
> >>
> >> It does indeed assume that, and a few other things. Currently policy
> >> doesn't specify what the download page URL should be, but once
>

Re: Proposal to add "release requirements" check

[Dave Fisher]

> On Nov 8, 2021, at 9:48 AM, Jarek Potiuk  wrote:
> 
>> However this does not solve the issue of stale download pages: how do
> you know when a release version is no longer supported?
> 
> What do you mean?
> 
> Is there a requirement for that that I missed? I believe the
> requirement is only about removing artifacts from "downloads.a.o" and
> this is a completely different issue.

There is no definitive rule about the path to the download page. There is no 
standard and with over 200 projects, 35 podlings, and lots of products change 
is difficult.

There is one definite requirement that can be checked. Whether the releases in 
svn are properly formed.

We do this in the Incubator with the Clutch Analysis. 
https://svn.apache.org/repos/asf/incubator/public/trunk/clutch2.py - L1132
Output found by clicking on a podling on https://incubator.apache.org/clutch/

We also have code in ASF Pelican - 
https://github.com/apache/infrastructure-pelican/blob/master/plugins/asfdata.py#L362
That was used to produce this example for Hadoop - 
https://template.staged.apache.org/downloads.html
Using this template - 
https://github.com/apache/template-site/blob/main/content/downloads.ezmd
Control is - https://github.com/apache/template-site/blob/main/asfdata.yaml#L110

To review projects you might find out where their download page is by reviewing 
the project DOAP file, but we’ve never been able to get all projects to 
maintain one.

Maybe it’s time to more carefully define what the ask is here.

> 
> On Mon, Nov 8, 2021 at 6:42 PM sebb  wrote:
>> 
>> On Mon, 8 Nov 2021 at 17:11, Jarek Potiuk  wrote:
>>> 
>>> Yeah. We will have different URLs especially as more artifacts are
>>> produced (airflow has 3 of those each with different download pages).
>>> 
>>> But this should be straightforward when you can add the url when
>>> registering the release (as Sebb proposed). According to the release
>>> process you should register each such "release type" separately and
>>> adding a URL there will do the job nicely. And initially it can be
>>> optional and once we figure out how to communicate it best and maybe
>>> try with some projects first we can make it obligatory at release
>>> registration. We could make it convenient - for example an easy way to
>>> copy a previously added URL from selected previous release or
>>> autocomplete from previous entries - making the process of entry as
>>> smooth (or even smoother) than it is today.
>> 
>> Or you could just ask for the URL if the release is not already on a
>> download page.
>> This would mean parsing the page of course, but could be used to
>> provide immediate feedback if the page is non-conforming.
>> 
>> Asking for the page might also nudge projects to move to generic pages
>> rather than one per release.
>> 
>> However this does not solve the issue of stale download pages: how do
>> you know when a release version is no longer supported?
>> 
>>> J.
>>> 
>>> 
>>> On Mon, Nov 8, 2021 at 4:19 PM sebb  wrote:
 
 On Mon, 8 Nov 2021 at 11:29, Hans Van Akelyen
  wrote:
> 
> Hi,
> 
> I think a certain level of uniformity would not be bad for this.
> Having a fixed path to the downloads for each project makes it simpler for
> the users to "shop" for Apache projects.
> 
> Having .a.o/download/ as a link that should point to the 
> resources
> makes it simpler for everyone.
 
 I agree, but expect a lot of complaints if you try to get this enforced.
 (and there's not much point unless it is enforced).
 
> Kind Regards,
> Hans
> 
> On Mon, 8 Nov 2021 at 12:19, Justin Mclean  
> wrote:
> 
>> Hi,
>> 
>>> However that assumes that you know where to find the download page(s).
>>> It also assumes that projects update their download page(s) to only
>>> show current releases.
>> 
>> It does indeed assume that, and a few other things. Currently policy
>> doesn't specify what the download page URL should be, but once known I
>> would guess it wouldn’t change often.
>> 
>> Kind Regards,
>> Justin
>> 
>> 
>> 
>> -
>> To unsubscribe, e-mail: dev-unsubscr...@community.apache.org
>> For additional commands, e-mail: dev-h...@community.apache.org
>> 
>> 
 
 -
 To unsubscribe, e-mail: dev-unsubscr...@community.apache.org
 For additional commands, e-mail: dev-h...@community.apache.org
 
>>> 
>>> -
>>> To unsubscribe, e-mail: dev-unsubscr...@community.apache.org
>>> For additional commands, e-mail: dev-h...@community.apache.org
>>> 
>> 
>> -
>> To unsubscribe, e-mail: dev-unsubscr...@community.apache.org
>> For additional commands, e-mail: dev

Re: Proposal to add "release requirements" check

[Jarek Potiuk]

> However this does not solve the issue of stale download pages: how do
you know when a release version is no longer supported?

What do you mean?

Is there a requirement for that that I missed? I believe the
requirement is only about removing artifacts from "downloads.a.o" and
this is a completely different issue.

On Mon, Nov 8, 2021 at 6:42 PM sebb  wrote:
>
> On Mon, 8 Nov 2021 at 17:11, Jarek Potiuk  wrote:
> >
> > Yeah. We will have different URLs especially as more artifacts are
> > produced (airflow has 3 of those each with different download pages).
> >
> > But this should be straightforward when you can add the url when
> > registering the release (as Sebb proposed). According to the release
> > process you should register each such "release type" separately and
> > adding a URL there will do the job nicely. And initially it can be
> > optional and once we figure out how to communicate it best and maybe
> > try with some projects first we can make it obligatory at release
> > registration. We could make it convenient - for example an easy way to
> > copy a previously added URL from selected previous release or
> > autocomplete from previous entries - making the process of entry as
> > smooth (or even smoother) than it is today.
>
> Or you could just ask for the URL if the release is not already on a
> download page.
> This would mean parsing the page of course, but could be used to
> provide immediate feedback if the page is non-conforming.
>
> Asking for the page might also nudge projects to move to generic pages
> rather than one per release.
>
> However this does not solve the issue of stale download pages: how do
> you know when a release version is no longer supported?
>
> > J.
> >
> >
> > On Mon, Nov 8, 2021 at 4:19 PM sebb  wrote:
> > >
> > > On Mon, 8 Nov 2021 at 11:29, Hans Van Akelyen
> > >  wrote:
> > > >
> > > > Hi,
> > > >
> > > > I think a certain level of uniformity would not be bad for this.
> > > > Having a fixed path to the downloads for each project makes it simpler 
> > > > for
> > > > the users to "shop" for Apache projects.
> > > >
> > > > Having .a.o/download/ as a link that should point to the 
> > > > resources
> > > > makes it simpler for everyone.
> > >
> > > I agree, but expect a lot of complaints if you try to get this enforced.
> > > (and there's not much point unless it is enforced).
> > >
> > > > Kind Regards,
> > > > Hans
> > > >
> > > > On Mon, 8 Nov 2021 at 12:19, Justin Mclean  
> > > > wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > > However that assumes that you know where to find the download 
> > > > > > page(s).
> > > > > > It also assumes that projects update their download page(s) to only
> > > > > > show current releases.
> > > > >
> > > > > It does indeed assume that, and a few other things. Currently policy
> > > > > doesn't specify what the download page URL should be, but once known I
> > > > > would guess it wouldn’t change often.
> > > > >
> > > > > Kind Regards,
> > > > > Justin
> > > > >
> > > > >
> > > > >
> > > > > -
> > > > > To unsubscribe, e-mail: dev-unsubscr...@community.apache.org
> > > > > For additional commands, e-mail: dev-h...@community.apache.org
> > > > >
> > > > >
> > >
> > > -
> > > To unsubscribe, e-mail: dev-unsubscr...@community.apache.org
> > > For additional commands, e-mail: dev-h...@community.apache.org
> > >
> >
> > -
> > To unsubscribe, e-mail: dev-unsubscr...@community.apache.org
> > For additional commands, e-mail: dev-h...@community.apache.org
> >
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@community.apache.org
> For additional commands, e-mail: dev-h...@community.apache.org
>

-
To unsubscribe, e-mail: dev-unsubscr...@community.apache.org
For additional commands, e-mail: dev-h...@community.apache.org

Re: Proposal to add "release requirements" check

[sebb]

On Mon, 8 Nov 2021 at 17:11, Jarek Potiuk  wrote:
>
> Yeah. We will have different URLs especially as more artifacts are
> produced (airflow has 3 of those each with different download pages).
>
> But this should be straightforward when you can add the url when
> registering the release (as Sebb proposed). According to the release
> process you should register each such "release type" separately and
> adding a URL there will do the job nicely. And initially it can be
> optional and once we figure out how to communicate it best and maybe
> try with some projects first we can make it obligatory at release
> registration. We could make it convenient - for example an easy way to
> copy a previously added URL from selected previous release or
> autocomplete from previous entries - making the process of entry as
> smooth (or even smoother) than it is today.

Or you could just ask for the URL if the release is not already on a
download page.
This would mean parsing the page of course, but could be used to
provide immediate feedback if the page is non-conforming.

Asking for the page might also nudge projects to move to generic pages
rather than one per release.

However this does not solve the issue of stale download pages: how do
you know when a release version is no longer supported?

> J.
>
>
> On Mon, Nov 8, 2021 at 4:19 PM sebb  wrote:
> >
> > On Mon, 8 Nov 2021 at 11:29, Hans Van Akelyen
> >  wrote:
> > >
> > > Hi,
> > >
> > > I think a certain level of uniformity would not be bad for this.
> > > Having a fixed path to the downloads for each project makes it simpler for
> > > the users to "shop" for Apache projects.
> > >
> > > Having .a.o/download/ as a link that should point to the 
> > > resources
> > > makes it simpler for everyone.
> >
> > I agree, but expect a lot of complaints if you try to get this enforced.
> > (and there's not much point unless it is enforced).
> >
> > > Kind Regards,
> > > Hans
> > >
> > > On Mon, 8 Nov 2021 at 12:19, Justin Mclean  
> > > wrote:
> > >
> > > > Hi,
> > > >
> > > > > However that assumes that you know where to find the download page(s).
> > > > > It also assumes that projects update their download page(s) to only
> > > > > show current releases.
> > > >
> > > > It does indeed assume that, and a few other things. Currently policy
> > > > doesn't specify what the download page URL should be, but once known I
> > > > would guess it wouldn’t change often.
> > > >
> > > > Kind Regards,
> > > > Justin
> > > >
> > > >
> > > >
> > > > -
> > > > To unsubscribe, e-mail: dev-unsubscr...@community.apache.org
> > > > For additional commands, e-mail: dev-h...@community.apache.org
> > > >
> > > >
> >
> > -
> > To unsubscribe, e-mail: dev-unsubscr...@community.apache.org
> > For additional commands, e-mail: dev-h...@community.apache.org
> >
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@community.apache.org
> For additional commands, e-mail: dev-h...@community.apache.org
>

-
To unsubscribe, e-mail: dev-unsubscr...@community.apache.org
For additional commands, e-mail: dev-h...@community.apache.org

Re: Proposal to add "release requirements" check

[Jarek Potiuk]

Yeah. We will have different URLs especially as more artifacts are
produced (airflow has 3 of those each with different download pages).

But this should be straightforward when you can add the url when
registering the release (as Sebb proposed). According to the release
process you should register each such "release type" separately and
adding a URL there will do the job nicely. And initially it can be
optional and once we figure out how to communicate it best and maybe
try with some projects first we can make it obligatory at release
registration. We could make it convenient - for example an easy way to
copy a previously added URL from selected previous release or
autocomplete from previous entries - making the process of entry as
smooth (or even smoother) than it is today.

J.


On Mon, Nov 8, 2021 at 4:19 PM sebb  wrote:
>
> On Mon, 8 Nov 2021 at 11:29, Hans Van Akelyen
>  wrote:
> >
> > Hi,
> >
> > I think a certain level of uniformity would not be bad for this.
> > Having a fixed path to the downloads for each project makes it simpler for
> > the users to "shop" for Apache projects.
> >
> > Having .a.o/download/ as a link that should point to the resources
> > makes it simpler for everyone.
>
> I agree, but expect a lot of complaints if you try to get this enforced.
> (and there's not much point unless it is enforced).
>
> > Kind Regards,
> > Hans
> >
> > On Mon, 8 Nov 2021 at 12:19, Justin Mclean  wrote:
> >
> > > Hi,
> > >
> > > > However that assumes that you know where to find the download page(s).
> > > > It also assumes that projects update their download page(s) to only
> > > > show current releases.
> > >
> > > It does indeed assume that, and a few other things. Currently policy
> > > doesn't specify what the download page URL should be, but once known I
> > > would guess it wouldn’t change often.
> > >
> > > Kind Regards,
> > > Justin
> > >
> > >
> > >
> > > -
> > > To unsubscribe, e-mail: dev-unsubscr...@community.apache.org
> > > For additional commands, e-mail: dev-h...@community.apache.org
> > >
> > >
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@community.apache.org
> For additional commands, e-mail: dev-h...@community.apache.org
>

-
To unsubscribe, e-mail: dev-unsubscr...@community.apache.org
For additional commands, e-mail: dev-h...@community.apache.org

Re: Proposal to add "release requirements" check

[sebb]

On Mon, 8 Nov 2021 at 11:29, Hans Van Akelyen
 wrote:
>
> Hi,
>
> I think a certain level of uniformity would not be bad for this.
> Having a fixed path to the downloads for each project makes it simpler for
> the users to "shop" for Apache projects.
>
> Having .a.o/download/ as a link that should point to the resources
> makes it simpler for everyone.

I agree, but expect a lot of complaints if you try to get this enforced.
(and there's not much point unless it is enforced).

> Kind Regards,
> Hans
>
> On Mon, 8 Nov 2021 at 12:19, Justin Mclean  wrote:
>
> > Hi,
> >
> > > However that assumes that you know where to find the download page(s).
> > > It also assumes that projects update their download page(s) to only
> > > show current releases.
> >
> > It does indeed assume that, and a few other things. Currently policy
> > doesn't specify what the download page URL should be, but once known I
> > would guess it wouldn’t change often.
> >
> > Kind Regards,
> > Justin
> >
> >
> >
> > -
> > To unsubscribe, e-mail: dev-unsubscr...@community.apache.org
> > For additional commands, e-mail: dev-h...@community.apache.org
> >
> >

-
To unsubscribe, e-mail: dev-unsubscr...@community.apache.org
For additional commands, e-mail: dev-h...@community.apache.org

Re: Proposal to add "release requirements" check

[sebb]

On Mon, 8 Nov 2021 at 11:19, Justin Mclean  wrote:
>
> Hi,
>
> > However that assumes that you know where to find the download page(s).
> > It also assumes that projects update their download page(s) to only
> > show current releases.
>
> It does indeed assume that, and a few other things. Currently policy doesn't 
> specify what the download page URL should be, but once known I would guess it 
> wouldn’t change often.

If only.

Several projects use a different URL for each release. For example
https://daffodil.apache.org/releases/
https://db.apache.org/derby/derby_downloads.html
https://guacamole.apache.org/releases/

Some projects use different pages for source and binaries, e.g.
https://ant.apache.org/bindownload.cgi

> Kind Regards,
> Justin
>
>
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@community.apache.org
> For additional commands, e-mail: dev-h...@community.apache.org
>

-
To unsubscribe, e-mail: dev-unsubscr...@community.apache.org
For additional commands, e-mail: dev-h...@community.apache.org

Re: Proposal to add "release requirements" check

[Hans Van Akelyen]

Hi,

I think a certain level of uniformity would not be bad for this.
Having a fixed path to the downloads for each project makes it simpler for
the users to "shop" for Apache projects.

Having .a.o/download/ as a link that should point to the resources
makes it simpler for everyone.

Kind Regards,
Hans

On Mon, 8 Nov 2021 at 12:19, Justin Mclean  wrote:

> Hi,
>
> > However that assumes that you know where to find the download page(s).
> > It also assumes that projects update their download page(s) to only
> > show current releases.
>
> It does indeed assume that, and a few other things. Currently policy
> doesn't specify what the download page URL should be, but once known I
> would guess it wouldn’t change often.
>
> Kind Regards,
> Justin
>
>
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@community.apache.org
> For additional commands, e-mail: dev-h...@community.apache.org
>
>

Re: Proposal to add "release requirements" check

[Justin Mclean]

Hi,

> However that assumes that you know where to find the download page(s).
> It also assumes that projects update their download page(s) to only
> show current releases.

It does indeed assume that, and a few other things. Currently policy doesn't 
specify what the download page URL should be, but once known I would guess it 
wouldn’t change often.

Kind Regards,
Justin



-
To unsubscribe, e-mail: dev-unsubscr...@community.apache.org
For additional commands, e-mail: dev-h...@community.apache.org

Re: Proposal to add "release requirements" check

[sebb]

On Mon, 8 Nov 2021 at 00:49, Justin Mclean  wrote:
>
> Hi,
>
> > Purging old releases needs at least two data sources:
> > - the list of files on downloads.apache.org  
> > (or the dist.a.o source).
> > This is trivial to obtain.
> > - the list of current versions for each release. This is the most
> > difficult part.
>
> The check dist area script I wrote attempts to do this. [1] It get the list 
> of released version from the download page and matches them up with what is 
> in the dist area.

However that assumes that you know where to find the download page(s).
It also assumes that projects update their download page(s) to only
show current releases.

Once you have that information, the rest is relatively easy (unless
the download page uses Javascript)

> To be complete you also need to look in the archives (which I’ve not done 
> yet).  It does have some of the issues you mention but it’s a start and 
> worked for abut 90% of Incubating projects. having something that works for 
> project project is probably a good start and it can be refined from there.
>
> Kind Regards,
> Justin
>
> 1. https://github.com/justinmclean/ReleaseChecker/blob/main/checkdistarea.py

-
To unsubscribe, e-mail: dev-unsubscr...@community.apache.org
For additional commands, e-mail: dev-h...@community.apache.org

Re: Proposal to add "release requirements" check

[Justin Mclean]

Hi,

> Purging old releases needs at least two data sources:
> - the list of files on downloads.apache.org  
> (or the dist.a.o source).
> This is trivial to obtain.
> - the list of current versions for each release. This is the most
> difficult part.

The check dist area script I wrote attempts to do this. [1] It get the list of 
released version from the download page and matches them up with what is in the 
dist area. To be complete you also need to look in the archives (which I’ve not 
done yet).  It does have some of the issues you mention but it’s a start and 
worked for abut 90% of Incubating projects. having something that works for 
project project is probably a good start and it can be refined from there.

Kind Regards,
Justin

1. https://github.com/justinmclean/ReleaseChecker/blob/main/checkdistarea.py

Re: Proposal to add "release requirements" check

[Jarek Potiuk]

Of course, I know there are differences, and yes I think we need at
least a few iterations and cooperation of different projects.

How I really like to work with such projects is that I roughly think
about a solution, then experiment, see what works and finally when I
have a working POC I involve people and we work together on finalizing
stuff. I strongly believe in  "Emergent design" and it is based on the
assumption that (especially in open-source) experiments, trying
different things and sometimes failing - but learning from that and
moving forward - is the best approach. I think this kind of project is
perfect for such approach - where you can gradually add groups of
projects that do things similarly and then incrementally add new
patterns.Taking into account that it is not "business critical and the
whole cycle of board reviews takes many moths, it seems ideal for such
approach.

This will undoubtedly take a while to complete and sort out all the
issues and since I am doing it in my free time, I hope I can engage
others - especially those who seem to be vitally interested in making
it work. I do not have "all answers" yet (or even I have almost none
now) and I hope not only myself but also people from other projects
can be involved and we can all learn something new. Having all answers
upfront is a bad idea for such kind of community project, where I hope
to engage others.

> By the way, how are you going to let people know about the changes to the 
> reporter tool?

Speaking of others being involved in the project- would you like to
help with that part?
I am happy to hear some suggestions for that part especially?
What do you think will work best? Any ideas there ?

J.

-
To unsubscribe, e-mail: dev-unsubscr...@community.apache.org
For additional commands, e-mail: dev-h...@community.apache.org

Re: Proposal to add "release requirements" check

[sebb]

On Sun, 7 Nov 2021 at 21:21, Jarek Potiuk  wrote:
>
> I (and Craig I guess) were discussing the  whole solution - which also
> includes cleaning download artifacts.

Purging old releases needs at least two data sources:
- the list of files on downloads.apache.org (or the dist.a.o source).
This is trivial to obtain.
- the list of current versions for each release. This is the most
difficult part.

Not only does it vary between PMCs, there are different release
strategies for different products.
For example Commons DBCP has multiple releases for different versions
of JDBC, whereas most other Commons components have one active
release.

Furthermore, the number of active releases for a product may change
from time to time.

Also products sometimes get retired.

I think you are going to need more than just a download page link to
enable accurate reporting of unpurged releases.

By the way, how are you going to let people know about the changes to
the reporter tool?

> J.
>
> On Sun, Nov 7, 2021 at 9:32 PM sebb  wrote:
> >
> > AFAICT there is only one case to handle.
> >
> > The reporter tool lists recent releases.
> > Each such release must have an associated download page.
> >
> > Whether there are multiple download pages or a shared one does not
> > matter in this context.
> >
> > On Sun, 7 Nov 2021 at 20:20, Jarek Potiuk  wrote:
> > >
> > > Yep. Those are all valid concerns and I will look into how we could
> > > handle those complexities. Hopefully there will be only a handful of
> > > various cases :D
> > >
> > > On Sun, Nov 7, 2021 at 9:17 PM Craig Russell  wrote:
> > > >
> > > > Hi Jarek,
> > > >
> > > > I think that updating the reporter tool to check for a conformant 
> > > > download(s) page is a great idea. I encourage you to see how to 
> > > > incorporate that check into the tool.
> > > >
> > > > I agree with you that sending out mass mailings is probably not going 
> > > > to result in changes in behavior. Only targeted messages are likely to 
> > > > work.
> > > >
> > > > One challenge will be to handle projects with multiple "independent" 
> > > > sub-projects like db that has jdo and derby with not much in common. 
> > > > They have completely independent web sites. So the tool will have to be 
> > > > told about all of the projects. The tool also needs to handle projects 
> > > > like Airflow which IIUC has multiple sub-projects that share a few 
> > > > common download pages and scripts.
> > > >
> > > > Warm regards,
> > > > Craig
> > > >
> > > > > On Nov 4, 2021, at 5:57 AM, Jarek Potiuk  wrote:
> > > > >
> > > > > Hello Everyone @devcom,
> > > > >
> > > > > I am following up after the discussion  at users@infra.a.o:
> > > > > https://lists.apache.org/thread/skswflvf0q8lbgkmxo18v19s62hkqf3j. 
> > > > > This is
> > > > > not a public mailing list and maybe not everyone here is subscribed 
> > > > > so let
> > > > > me summarise the context and the proposal. BTW. That's my view on the 
> > > > > state
> > > > > of the discussion we had and conclusions/proposals I drew from that.
> > > > >
> > > > > Context:
> > > > >
> > > > > We seem to have some of the projects that do not follow all the 
> > > > > important
> > > > > requirements when it comes to publishing GA releases for their 
> > > > > projects.
> > > > > The requirements are pretty firm and documented here:
> > > > > https://www.apache.org/legal/release-policy.html#host-GA and in the
> > > > > following paragraphs. They boil down to :
> > > > >
> > > > > a) have an installation page where you can verify releases with:
> > > > > * signatures and checksums (links to those should be directly do
> > > > > downloads.apache.org
> > > > > * packages - links to those should use closer.lua scripts (which are 
> > > > > then
> > > > > automatically mapped to CDN currently - previously to mirroring sites)
> > > > >
> > > > > b) Have they releases archived appropriately (i.e. only last version 
> > > > > from
> > > > > the development "branch"  should remain in downloads@a.o
> > > > >
> > > > > Proposal:
> > > > >
> > > > > I think the best way to get this solved permanently (rather than 
> > > > > sending
> > > > > reminder emails to everyone) is to add a check in the reporter tool (
> > > > > https://reporter.apache.org/about.html) to verify those two aspects, 
> > > > > so
> > > > > that they could be verified by projects themselves and the board  - 
> > > > > seems
> > > > > that this is a very important aspect that ASF is very concerned 
> > > > > about, so
> > > > > making it part of the board review makes sense and gives the 
> > > > > opportunity of
> > > > > individual flagging when the project does not follow it, It has 
> > > > > happened
> > > > > with our project - Airflow at the last board review, and we fixed all 
> > > > > that,
> > > > > but the idea is that the ASF has an easy to use tool that will add 
> > > > > such
> > > > > checks and flags
> > > > >
> > > > > Point b) is quite easy it seems. Airflow already has simple Python

Re: Proposal to add "release requirements" check

[Jarek Potiuk]

I (and Craig I guess) were discussing the  whole solution - which also
includes cleaning download artifacts. You incorrectly assumed that
this thread is only about the release page (see the original message
of mine). However, thanks for being as usual vigilant and pointing out
every possibly inaccurate statement.

J.

On Sun, Nov 7, 2021 at 9:32 PM sebb  wrote:
>
> AFAICT there is only one case to handle.
>
> The reporter tool lists recent releases.
> Each such release must have an associated download page.
>
> Whether there are multiple download pages or a shared one does not
> matter in this context.
>
> On Sun, 7 Nov 2021 at 20:20, Jarek Potiuk  wrote:
> >
> > Yep. Those are all valid concerns and I will look into how we could
> > handle those complexities. Hopefully there will be only a handful of
> > various cases :D
> >
> > On Sun, Nov 7, 2021 at 9:17 PM Craig Russell  wrote:
> > >
> > > Hi Jarek,
> > >
> > > I think that updating the reporter tool to check for a conformant 
> > > download(s) page is a great idea. I encourage you to see how to 
> > > incorporate that check into the tool.
> > >
> > > I agree with you that sending out mass mailings is probably not going to 
> > > result in changes in behavior. Only targeted messages are likely to work.
> > >
> > > One challenge will be to handle projects with multiple "independent" 
> > > sub-projects like db that has jdo and derby with not much in common. They 
> > > have completely independent web sites. So the tool will have to be told 
> > > about all of the projects. The tool also needs to handle projects like 
> > > Airflow which IIUC has multiple sub-projects that share a few common 
> > > download pages and scripts.
> > >
> > > Warm regards,
> > > Craig
> > >
> > > > On Nov 4, 2021, at 5:57 AM, Jarek Potiuk  wrote:
> > > >
> > > > Hello Everyone @devcom,
> > > >
> > > > I am following up after the discussion  at users@infra.a.o:
> > > > https://lists.apache.org/thread/skswflvf0q8lbgkmxo18v19s62hkqf3j. This 
> > > > is
> > > > not a public mailing list and maybe not everyone here is subscribed so 
> > > > let
> > > > me summarise the context and the proposal. BTW. That's my view on the 
> > > > state
> > > > of the discussion we had and conclusions/proposals I drew from that.
> > > >
> > > > Context:
> > > >
> > > > We seem to have some of the projects that do not follow all the 
> > > > important
> > > > requirements when it comes to publishing GA releases for their projects.
> > > > The requirements are pretty firm and documented here:
> > > > https://www.apache.org/legal/release-policy.html#host-GA and in the
> > > > following paragraphs. They boil down to :
> > > >
> > > > a) have an installation page where you can verify releases with:
> > > > * signatures and checksums (links to those should be directly do
> > > > downloads.apache.org
> > > > * packages - links to those should use closer.lua scripts (which are 
> > > > then
> > > > automatically mapped to CDN currently - previously to mirroring sites)
> > > >
> > > > b) Have they releases archived appropriately (i.e. only last version 
> > > > from
> > > > the development "branch"  should remain in downloads@a.o
> > > >
> > > > Proposal:
> > > >
> > > > I think the best way to get this solved permanently (rather than sending
> > > > reminder emails to everyone) is to add a check in the reporter tool (
> > > > https://reporter.apache.org/about.html) to verify those two aspects, so
> > > > that they could be verified by projects themselves and the board  - 
> > > > seems
> > > > that this is a very important aspect that ASF is very concerned about, 
> > > > so
> > > > making it part of the board review makes sense and gives the 
> > > > opportunity of
> > > > individual flagging when the project does not follow it, It has happened
> > > > with our project - Airflow at the last board review, and we fixed all 
> > > > that,
> > > > but the idea is that the ASF has an easy to use tool that will add such
> > > > checks and flags
> > > >
> > > > Point b) is quite easy it seems. Airflow already has simple Python 
> > > > script
> > > > that does that can be adjusted or rewritten to handle multiple "lines" 
> > > > of
> > > > development (with some per-project specifics likely) :
> > > > https://github.com/apache/airflow/blob/main/dev/provider_packages/remove_old_releases.py.
> > > > It can also be used to actually help with clean-up (we use it for that
> > > > purpose as we have ~40/50 packages to release every month).
> > > >
> > > > Point  a) also should not be difficult as long as each project will 
> > > > submit
> > > > their installation page links, we can scrape them and see if the pages
> > > > contain signature/ascii direct links and closer.lua links for packages.
> > > >
> > > > Do you think it is a good idea? I just wanted to run it by the 
> > > > community to
> > > > see what you think of it, whether you see any potential
> > > > problems/limitations/concerns?
> > > >
> > > > Just one

Re: Proposal to add "release requirements" check

[sebb]

AFAICT there is only one case to handle.

The reporter tool lists recent releases.
Each such release must have an associated download page.

Whether there are multiple download pages or a shared one does not
matter in this context.

On Sun, 7 Nov 2021 at 20:20, Jarek Potiuk  wrote:
>
> Yep. Those are all valid concerns and I will look into how we could
> handle those complexities. Hopefully there will be only a handful of
> various cases :D
>
> On Sun, Nov 7, 2021 at 9:17 PM Craig Russell  wrote:
> >
> > Hi Jarek,
> >
> > I think that updating the reporter tool to check for a conformant 
> > download(s) page is a great idea. I encourage you to see how to incorporate 
> > that check into the tool.
> >
> > I agree with you that sending out mass mailings is probably not going to 
> > result in changes in behavior. Only targeted messages are likely to work.
> >
> > One challenge will be to handle projects with multiple "independent" 
> > sub-projects like db that has jdo and derby with not much in common. They 
> > have completely independent web sites. So the tool will have to be told 
> > about all of the projects. The tool also needs to handle projects like 
> > Airflow which IIUC has multiple sub-projects that share a few common 
> > download pages and scripts.
> >
> > Warm regards,
> > Craig
> >
> > > On Nov 4, 2021, at 5:57 AM, Jarek Potiuk  wrote:
> > >
> > > Hello Everyone @devcom,
> > >
> > > I am following up after the discussion  at users@infra.a.o:
> > > https://lists.apache.org/thread/skswflvf0q8lbgkmxo18v19s62hkqf3j. This is
> > > not a public mailing list and maybe not everyone here is subscribed so let
> > > me summarise the context and the proposal. BTW. That's my view on the 
> > > state
> > > of the discussion we had and conclusions/proposals I drew from that.
> > >
> > > Context:
> > >
> > > We seem to have some of the projects that do not follow all the important
> > > requirements when it comes to publishing GA releases for their projects.
> > > The requirements are pretty firm and documented here:
> > > https://www.apache.org/legal/release-policy.html#host-GA and in the
> > > following paragraphs. They boil down to :
> > >
> > > a) have an installation page where you can verify releases with:
> > > * signatures and checksums (links to those should be directly do
> > > downloads.apache.org
> > > * packages - links to those should use closer.lua scripts (which are then
> > > automatically mapped to CDN currently - previously to mirroring sites)
> > >
> > > b) Have they releases archived appropriately (i.e. only last version from
> > > the development "branch"  should remain in downloads@a.o
> > >
> > > Proposal:
> > >
> > > I think the best way to get this solved permanently (rather than sending
> > > reminder emails to everyone) is to add a check in the reporter tool (
> > > https://reporter.apache.org/about.html) to verify those two aspects, so
> > > that they could be verified by projects themselves and the board  - seems
> > > that this is a very important aspect that ASF is very concerned about, so
> > > making it part of the board review makes sense and gives the opportunity 
> > > of
> > > individual flagging when the project does not follow it, It has happened
> > > with our project - Airflow at the last board review, and we fixed all 
> > > that,
> > > but the idea is that the ASF has an easy to use tool that will add such
> > > checks and flags
> > >
> > > Point b) is quite easy it seems. Airflow already has simple Python script
> > > that does that can be adjusted or rewritten to handle multiple "lines" of
> > > development (with some per-project specifics likely) :
> > > https://github.com/apache/airflow/blob/main/dev/provider_packages/remove_old_releases.py.
> > > It can also be used to actually help with clean-up (we use it for that
> > > purpose as we have ~40/50 packages to release every month).
> > >
> > > Point  a) also should not be difficult as long as each project will submit
> > > their installation page links, we can scrape them and see if the pages
> > > contain signature/ascii direct links and closer.lua links for packages.
> > >
> > > Do you think it is a good idea? I just wanted to run it by the community 
> > > to
> > > see what you think of it, whether you see any potential
> > > problems/limitations/concerns?
> > >
> > > Just one request please - without yet getting into technical details and
> > > discussing HOWs. We will have time for that.
> > >
> > > If there is a community (and board :) buy-in, I am happy to
> > > implement/lead/introduce it. It would dramatically help if others -
> > > especially people from various projects that might have some special cases
> > > - could help with testing/piloting it.
> > >
> > > J.
> >
> > Craig L Russell
> > c...@apache.org
> >
> >
> > -
> > To unsubscribe, e-mail: dev-unsubscr...@community.apache.org
> > For additional commands, e-mail: dev-h...@com

Re: Proposal to add "release requirements" check

[Jarek Potiuk]

Yep. Those are all valid concerns and I will look into how we could
handle those complexities. Hopefully there will be only a handful of
various cases :D

On Sun, Nov 7, 2021 at 9:17 PM Craig Russell  wrote:
>
> Hi Jarek,
>
> I think that updating the reporter tool to check for a conformant download(s) 
> page is a great idea. I encourage you to see how to incorporate that check 
> into the tool.
>
> I agree with you that sending out mass mailings is probably not going to 
> result in changes in behavior. Only targeted messages are likely to work.
>
> One challenge will be to handle projects with multiple "independent" 
> sub-projects like db that has jdo and derby with not much in common. They 
> have completely independent web sites. So the tool will have to be told about 
> all of the projects. The tool also needs to handle projects like Airflow 
> which IIUC has multiple sub-projects that share a few common download pages 
> and scripts.
>
> Warm regards,
> Craig
>
> > On Nov 4, 2021, at 5:57 AM, Jarek Potiuk  wrote:
> >
> > Hello Everyone @devcom,
> >
> > I am following up after the discussion  at users@infra.a.o:
> > https://lists.apache.org/thread/skswflvf0q8lbgkmxo18v19s62hkqf3j. This is
> > not a public mailing list and maybe not everyone here is subscribed so let
> > me summarise the context and the proposal. BTW. That's my view on the state
> > of the discussion we had and conclusions/proposals I drew from that.
> >
> > Context:
> >
> > We seem to have some of the projects that do not follow all the important
> > requirements when it comes to publishing GA releases for their projects.
> > The requirements are pretty firm and documented here:
> > https://www.apache.org/legal/release-policy.html#host-GA and in the
> > following paragraphs. They boil down to :
> >
> > a) have an installation page where you can verify releases with:
> > * signatures and checksums (links to those should be directly do
> > downloads.apache.org
> > * packages - links to those should use closer.lua scripts (which are then
> > automatically mapped to CDN currently - previously to mirroring sites)
> >
> > b) Have they releases archived appropriately (i.e. only last version from
> > the development "branch"  should remain in downloads@a.o
> >
> > Proposal:
> >
> > I think the best way to get this solved permanently (rather than sending
> > reminder emails to everyone) is to add a check in the reporter tool (
> > https://reporter.apache.org/about.html) to verify those two aspects, so
> > that they could be verified by projects themselves and the board  - seems
> > that this is a very important aspect that ASF is very concerned about, so
> > making it part of the board review makes sense and gives the opportunity of
> > individual flagging when the project does not follow it, It has happened
> > with our project - Airflow at the last board review, and we fixed all that,
> > but the idea is that the ASF has an easy to use tool that will add such
> > checks and flags
> >
> > Point b) is quite easy it seems. Airflow already has simple Python script
> > that does that can be adjusted or rewritten to handle multiple "lines" of
> > development (with some per-project specifics likely) :
> > https://github.com/apache/airflow/blob/main/dev/provider_packages/remove_old_releases.py.
> > It can also be used to actually help with clean-up (we use it for that
> > purpose as we have ~40/50 packages to release every month).
> >
> > Point  a) also should not be difficult as long as each project will submit
> > their installation page links, we can scrape them and see if the pages
> > contain signature/ascii direct links and closer.lua links for packages.
> >
> > Do you think it is a good idea? I just wanted to run it by the community to
> > see what you think of it, whether you see any potential
> > problems/limitations/concerns?
> >
> > Just one request please - without yet getting into technical details and
> > discussing HOWs. We will have time for that.
> >
> > If there is a community (and board :) buy-in, I am happy to
> > implement/lead/introduce it. It would dramatically help if others -
> > especially people from various projects that might have some special cases
> > - could help with testing/piloting it.
> >
> > J.
>
> Craig L Russell
> c...@apache.org
>
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@community.apache.org
> For additional commands, e-mail: dev-h...@community.apache.org
>

-
To unsubscribe, e-mail: dev-unsubscr...@community.apache.org
For additional commands, e-mail: dev-h...@community.apache.org

Re: Proposal to add "release requirements" check

[Craig Russell]

Hi Jarek,

I think that updating the reporter tool to check for a conformant download(s) 
page is a great idea. I encourage you to see how to incorporate that check into 
the tool. 

I agree with you that sending out mass mailings is probably not going to result 
in changes in behavior. Only targeted messages are likely to work.

One challenge will be to handle projects with multiple "independent" 
sub-projects like db that has jdo and derby with not much in common. They have 
completely independent web sites. So the tool will have to be told about all of 
the projects. The tool also needs to handle projects like Airflow which IIUC 
has multiple sub-projects that share a few common download pages and scripts.

Warm regards,
Craig

> On Nov 4, 2021, at 5:57 AM, Jarek Potiuk  wrote:
> 
> Hello Everyone @devcom,
> 
> I am following up after the discussion  at users@infra.a.o:
> https://lists.apache.org/thread/skswflvf0q8lbgkmxo18v19s62hkqf3j. This is
> not a public mailing list and maybe not everyone here is subscribed so let
> me summarise the context and the proposal. BTW. That's my view on the state
> of the discussion we had and conclusions/proposals I drew from that.
> 
> Context:
> 
> We seem to have some of the projects that do not follow all the important
> requirements when it comes to publishing GA releases for their projects.
> The requirements are pretty firm and documented here:
> https://www.apache.org/legal/release-policy.html#host-GA and in the
> following paragraphs. They boil down to :
> 
> a) have an installation page where you can verify releases with:
> * signatures and checksums (links to those should be directly do
> downloads.apache.org
> * packages - links to those should use closer.lua scripts (which are then
> automatically mapped to CDN currently - previously to mirroring sites)
> 
> b) Have they releases archived appropriately (i.e. only last version from
> the development "branch"  should remain in downloads@a.o
> 
> Proposal:
> 
> I think the best way to get this solved permanently (rather than sending
> reminder emails to everyone) is to add a check in the reporter tool (
> https://reporter.apache.org/about.html) to verify those two aspects, so
> that they could be verified by projects themselves and the board  - seems
> that this is a very important aspect that ASF is very concerned about, so
> making it part of the board review makes sense and gives the opportunity of
> individual flagging when the project does not follow it, It has happened
> with our project - Airflow at the last board review, and we fixed all that,
> but the idea is that the ASF has an easy to use tool that will add such
> checks and flags
> 
> Point b) is quite easy it seems. Airflow already has simple Python script
> that does that can be adjusted or rewritten to handle multiple "lines" of
> development (with some per-project specifics likely) :
> https://github.com/apache/airflow/blob/main/dev/provider_packages/remove_old_releases.py.
> It can also be used to actually help with clean-up (we use it for that
> purpose as we have ~40/50 packages to release every month).
> 
> Point  a) also should not be difficult as long as each project will submit
> their installation page links, we can scrape them and see if the pages
> contain signature/ascii direct links and closer.lua links for packages.
> 
> Do you think it is a good idea? I just wanted to run it by the community to
> see what you think of it, whether you see any potential
> problems/limitations/concerns?
> 
> Just one request please - without yet getting into technical details and
> discussing HOWs. We will have time for that.
> 
> If there is a community (and board :) buy-in, I am happy to
> implement/lead/introduce it. It would dramatically help if others -
> especially people from various projects that might have some special cases
> - could help with testing/piloting it.
> 
> J.

Craig L Russell
c...@apache.org


-
To unsubscribe, e-mail: dev-unsubscr...@community.apache.org
For additional commands, e-mail: dev-h...@community.apache.org

Re: Proposal to add "release requirements" check

[Jarek Potiuk]

Hello, anyone else has an opinion :) ?

On Fri, Nov 5, 2021 at 12:04 AM Jarek Potiuk  wrote:
>
> Oh cool :)
>
> On Fri, Nov 5, 2021 at 12:01 AM Justin Mclean  
> wrote:
> >
> > Hi,
> >
> > I've written some Python scripts to check download pages and release areas 
> > for incubating projects. [1] They work in most cases if you know the 
> > download URL and the project has a single product. You can find them here - 
> > feel free to use or suggest changes.
> >
> > Kind Regards,
> > Justin
> >
> > 1. https://github.com/justinmclean/ReleaseChecker
> > -
> > To unsubscribe, e-mail: dev-unsubscr...@community.apache.org
> > For additional commands, e-mail: dev-h...@community.apache.org
> >

-
To unsubscribe, e-mail: dev-unsubscr...@community.apache.org
For additional commands, e-mail: dev-h...@community.apache.org

Re: Proposal to add "release requirements" check

[Jarek Potiuk]

Oh cool :)

On Fri, Nov 5, 2021 at 12:01 AM Justin Mclean  wrote:
>
> Hi,
>
> I've written some Python scripts to check download pages and release areas 
> for incubating projects. [1] They work in most cases if you know the download 
> URL and the project has a single product. You can find them here - feel free 
> to use or suggest changes.
>
> Kind Regards,
> Justin
>
> 1. https://github.com/justinmclean/ReleaseChecker
> -
> To unsubscribe, e-mail: dev-unsubscr...@community.apache.org
> For additional commands, e-mail: dev-h...@community.apache.org
>

-
To unsubscribe, e-mail: dev-unsubscr...@community.apache.org
For additional commands, e-mail: dev-h...@community.apache.org

Re: Proposal to add "release requirements" check

[Justin Mclean]

Hi,

I've written some Python scripts to check download pages and release areas for 
incubating projects. [1] They work in most cases if you know the download URL 
and the project has a single product. You can find them here - feel free to use 
or suggest changes.

Kind Regards,
Justin

1. https://github.com/justinmclean/ReleaseChecker
-
To unsubscribe, e-mail: dev-unsubscr...@community.apache.org
For additional commands, e-mail: dev-h...@community.apache.org

Proposal to add "release requirements" check

[Jarek Potiuk]

Hello Everyone @devcom,

I am following up after the discussion  at users@infra.a.o:
https://lists.apache.org/thread/skswflvf0q8lbgkmxo18v19s62hkqf3j. This is
not a public mailing list and maybe not everyone here is subscribed so let
me summarise the context and the proposal. BTW. That's my view on the state
of the discussion we had and conclusions/proposals I drew from that.

Context:

We seem to have some of the projects that do not follow all the important
requirements when it comes to publishing GA releases for their projects.
The requirements are pretty firm and documented here:
https://www.apache.org/legal/release-policy.html#host-GA and in the
following paragraphs. They boil down to :

a) have an installation page where you can verify releases with:
* signatures and checksums (links to those should be directly do
downloads.apache.org
* packages - links to those should use closer.lua scripts (which are then
automatically mapped to CDN currently - previously to mirroring sites)

b) Have they releases archived appropriately (i.e. only last version from
the development "branch"  should remain in downloads@a.o

Proposal:

I think the best way to get this solved permanently (rather than sending
reminder emails to everyone) is to add a check in the reporter tool (
https://reporter.apache.org/about.html) to verify those two aspects, so
that they could be verified by projects themselves and the board  - seems
that this is a very important aspect that ASF is very concerned about, so
making it part of the board review makes sense and gives the opportunity of
individual flagging when the project does not follow it, It has happened
with our project - Airflow at the last board review, and we fixed all that,
but the idea is that the ASF has an easy to use tool that will add such
checks and flags

Point b) is quite easy it seems. Airflow already has simple Python script
that does that can be adjusted or rewritten to handle multiple "lines" of
development (with some per-project specifics likely) :
https://github.com/apache/airflow/blob/main/dev/provider_packages/remove_old_releases.py.
It can also be used to actually help with clean-up (we use it for that
purpose as we have ~40/50 packages to release every month).

Point  a) also should not be difficult as long as each project will submit
their installation page links, we can scrape them and see if the pages
contain signature/ascii direct links and closer.lua links for packages.

Do you think it is a good idea? I just wanted to run it by the community to
see what you think of it, whether you see any potential
problems/limitations/concerns?

Just one request please - without yet getting into technical details and
discussing HOWs. We will have time for that.

If there is a community (and board :) buy-in, I am happy to
implement/lead/introduce it. It would dramatically help if others -
especially people from various projects that might have some special cases
- could help with testing/piloting it.

J.