Re: Migration and consolidation helm charts for ASF projects from helm/charts to apache/charts git

[Jarek Potiuk]

Hello, 

I finally had some time after Airlfow 2.0 release and I opened discussion about 
the policy in legal-disc...@apache.org:
 
https://lists.apache.org/thread.html/r7c9ceb3d6c764119b14dfedb0e22957993d93cf529792c402aaa05fc%40%3Clegal-discuss.apache.org%3E

I propose we continue the discussion there:

On 2020/09/04 02:01:05, Ween Jiann  wrote: 
> Hi Kamil,
> 
> What's your take of having the helm chart in a single repo?
> It will lessen the load to maintain tests and distribution.
> 
> On 2020/09/03 19:54:17, Kamil Breguła  wrote: 
> > Hello,
> > 
> > In the case of Airflow, we have developed our own chart and want to develop
> > and release it soon. For now, there is little chance that our community
> > will be interested in developing two independent charts.
> > https://github.com/apache/airflow/issues/10523
> > 
> > That would be the 3rd method of deploying Airflow on Kubernetes maintained
> > by our project. We have Kubernetes Operators also.
> > https://github.com/apache/airflow-on-k8s-operator
> > 
> > Best regards,
> > Kamil Breguła
> > 
> > On Thu, Sep 3, 2020 at 9:33 PM Ween Jiann  wrote:
> > 
> > > Hi all,
> > >
> > > The helm team has deprecated the charts repository and will be archiving
> > > it in Nov. Here’s the link to the notice
> > > https://github.com/helm/charts/blob/master/README.md#deprecation-timeline
> > >
> > > It looks like that are quite a few charts for ASF projects such as
> > > Airflow, Hadoop, Spark, Solr (not exhaustive) that have yet to be adopted
> > > by anyone. It would make sense to re-home them into a single git in for
> > > example apache/charts. This would facilitate easier distribution of 
> > > various
> > > ASF projects via a single helm repo, and setting up tests and CI
> > > integration needs only to be done once.
> > >
> > > Cheers,
> > > WJ
> > >
> > > -
> > > To unsubscribe, e-mail: dev-unsubscr...@community.apache.org
> > > For additional commands, e-mail: dev-h...@community.apache.org
> > >
> > >
> > 
> 
> -
> To unsubscribe, e-mail: dev-unsubscr...@community.apache.org
> For additional commands, e-mail: dev-h...@community.apache.org
> 
> 

-
To unsubscribe, e-mail: dev-unsubscr...@community.apache.org
For additional commands, e-mail: dev-h...@community.apache.org

Re: Migration and consolidation helm charts for ASF projects from helm/charts to apache/charts git

[Jarek Potiuk]

Thanks Kaxil! I agree adding scripts should not be compulsory - I already
changed it to "best practice" wherever I mentioned it. If you think even
this is too much - I am happy to discuss :).

I think that really the definition of the packaging that would make
it possible to legally comply with the requirements is the most important
part of the proposal - all the rest is just clarification of intentions.

For the rebuilds - I think the most important is the intention that the
user SHOULD be able to rebuild it and should know how to do it. This is the
only really important bit. Whether those are scripts or some other ways -
it should be up to the project for sure. For many of those scripts or
automation might simply be impossible.

J.

On Sun, Oct 18, 2020 at 6:34 PM Kaxil Naik  wrote:

> I have added comments on the proposed document. I don't think we should
> make it compulsory to include scripts to rebuild binaries used in the
> projects. It
> is a maintenance nightmare.
>
> I agree though that we need a proper definition for the "convenience
> packages"
> and some guidelines around Docker Image and Helm Chart.
>
> Regards,
> Kaxil
> Apache Airflow PMC
>
> On Sun, Oct 18, 2020 at 3:14 PM Jarek Potiuk  wrote:
>
> > @Matt Sicker  - absolutely, thanks for mentioning
> that.
> > My intention is to hash-out as many details as possible and get as much
> > input as possible from relevant people to make a strong case to the
> board.
> > While I am not an ASF member myself, just a PMC, I watched some of the
> > Apache Con talks and I understand that board operates in asynchronous
> mode
> > and that they should get an information that they could digest earlier,
> > offline and make decisions quickly, so I want to make everything to make
> it
> > clear and straightforward and - possibly have a strong case, supported,
> or
> > at least with very limited number of strong "no" opinions, so I am still
> in
> > the phase of involving people and gathering feedback.
> >
> > @Justin  - I read it and I think this is all really great. I think it
> very
> > nicely documents the current policies and beyond, there is only one thing
> > in both - current release policies and the incubator distribution
> > guidelines that are a bit contradictory and I hope part of my proposal
> for
> > policy update addresses. And I know it's the toughest one.
> > It's about category X software inclusion in binary convenience packages.
> As
> > I see it is pretty much impossible not to include category X software in
> a
> > docker image. Since they are Linux-kernel based usually there are always
> > some GPL licensed libraries included in those.
> >
> > While I perfectly understand that "COMPILED PACKAGES" should not contain
> > Category X software, I think "Docker images" (and likely other binary
> > distribution mechanisms - for example those that package software for
> > installation on Windows/Mac workstation) can - and it should be allowed.
> > Unfortunately according to the policies "Convenience binaries need to
> > follow licensing policy and not include any category X licensed
> software."
> >
> > So the really only thing I am looking for is to introduce a separate
> > category for those different "packaging mechanism" and separate them out
> > (and explicitly name as category). For now we either release "software"
> or
> > "convenience binaries". And docker images are neither.
> >
> > J,
> >
> >
> >
> >
> > On Sun, Oct 18, 2020 at 3:14 AM Justin Mclean 
> > wrote:
> >
> > > Hi,
> > >
> > > > I am not even sure myself if I am the only one who feels the
> > > disconnection
> > > > between reality and the current policies, that's why I started the
> > thread
> > > > here.
> > >
> > > You would not be the only person who set this. People and project are
> > > making small steps to correct this. Recently Infra’s distribution
> policy
> > > was updated and the Incubator made these guidelines. [1] Note that
> Infra
> > do
> > > support docker as a platform.
> > >
> > > Thanks,
> > > Justin
> > >
> > > 1. https://incubator.apache.org/guides/distribution.html
> > > -
> > > To unsubscribe, e-mail: dev-unsubscr...@community.apache.org
> > > For additional commands, e-mail: dev-h...@community.apache.org
> > >
> > >
> >
> > --
> > +48 660 796 129
> >
>


-- 
+48 660 796 129

Re: Migration and consolidation helm charts for ASF projects from helm/charts to apache/charts git

[Kaxil Naik]

I have added comments on the proposed document. I don't think we should
make it compulsory to include scripts to rebuild binaries used in the
projects. It
is a maintenance nightmare.

I agree though that we need a proper definition for the "convenience
packages"
and some guidelines around Docker Image and Helm Chart.

Regards,
Kaxil
Apache Airflow PMC

On Sun, Oct 18, 2020 at 3:14 PM Jarek Potiuk  wrote:

> @Matt Sicker  - absolutely, thanks for mentioning that.
> My intention is to hash-out as many details as possible and get as much
> input as possible from relevant people to make a strong case to the board.
> While I am not an ASF member myself, just a PMC, I watched some of the
> Apache Con talks and I understand that board operates in asynchronous mode
> and that they should get an information that they could digest earlier,
> offline and make decisions quickly, so I want to make everything to make it
> clear and straightforward and - possibly have a strong case, supported, or
> at least with very limited number of strong "no" opinions, so I am still in
> the phase of involving people and gathering feedback.
>
> @Justin  - I read it and I think this is all really great. I think it very
> nicely documents the current policies and beyond, there is only one thing
> in both - current release policies and the incubator distribution
> guidelines that are a bit contradictory and I hope part of my proposal for
> policy update addresses. And I know it's the toughest one.
> It's about category X software inclusion in binary convenience packages. As
> I see it is pretty much impossible not to include category X software in a
> docker image. Since they are Linux-kernel based usually there are always
> some GPL licensed libraries included in those.
>
> While I perfectly understand that "COMPILED PACKAGES" should not contain
> Category X software, I think "Docker images" (and likely other binary
> distribution mechanisms - for example those that package software for
> installation on Windows/Mac workstation) can - and it should be allowed.
> Unfortunately according to the policies "Convenience binaries need to
> follow licensing policy and not include any category X licensed software."
>
> So the really only thing I am looking for is to introduce a separate
> category for those different "packaging mechanism" and separate them out
> (and explicitly name as category). For now we either release "software" or
> "convenience binaries". And docker images are neither.
>
> J,
>
>
>
>
> On Sun, Oct 18, 2020 at 3:14 AM Justin Mclean 
> wrote:
>
> > Hi,
> >
> > > I am not even sure myself if I am the only one who feels the
> > disconnection
> > > between reality and the current policies, that's why I started the
> thread
> > > here.
> >
> > You would not be the only person who set this. People and project are
> > making small steps to correct this. Recently Infra’s distribution policy
> > was updated and the Incubator made these guidelines. [1] Note that Infra
> do
> > support docker as a platform.
> >
> > Thanks,
> > Justin
> >
> > 1. https://incubator.apache.org/guides/distribution.html
> > -
> > To unsubscribe, e-mail: dev-unsubscr...@community.apache.org
> > For additional commands, e-mail: dev-h...@community.apache.org
> >
> >
>
> --
> +48 660 796 129
>

Re: Migration and consolidation helm charts for ASF projects from helm/charts to apache/charts git

[Jarek Potiuk]

@Matt Sicker  - absolutely, thanks for mentioning that.
My intention is to hash-out as many details as possible and get as much
input as possible from relevant people to make a strong case to the board.
While I am not an ASF member myself, just a PMC, I watched some of the
Apache Con talks and I understand that board operates in asynchronous mode
and that they should get an information that they could digest earlier,
offline and make decisions quickly, so I want to make everything to make it
clear and straightforward and - possibly have a strong case, supported, or
at least with very limited number of strong "no" opinions, so I am still in
the phase of involving people and gathering feedback.

@Justin  - I read it and I think this is all really great. I think it very
nicely documents the current policies and beyond, there is only one thing
in both - current release policies and the incubator distribution
guidelines that are a bit contradictory and I hope part of my proposal for
policy update addresses. And I know it's the toughest one.
It's about category X software inclusion in binary convenience packages. As
I see it is pretty much impossible not to include category X software in a
docker image. Since they are Linux-kernel based usually there are always
some GPL licensed libraries included in those.

While I perfectly understand that "COMPILED PACKAGES" should not contain
Category X software, I think "Docker images" (and likely other binary
distribution mechanisms - for example those that package software for
installation on Windows/Mac workstation) can - and it should be allowed.
Unfortunately according to the policies "Convenience binaries need to
follow licensing policy and not include any category X licensed software."

So the really only thing I am looking for is to introduce a separate
category for those different "packaging mechanism" and separate them out
(and explicitly name as category). For now we either release "software" or
"convenience binaries". And docker images are neither.

J,




On Sun, Oct 18, 2020 at 3:14 AM Justin Mclean 
wrote:

> Hi,
>
> > I am not even sure myself if I am the only one who feels the
> disconnection
> > between reality and the current policies, that's why I started the thread
> > here.
>
> You would not be the only person who set this. People and project are
> making small steps to correct this. Recently Infra’s distribution policy
> was updated and the Incubator made these guidelines. [1] Note that Infra do
> support docker as a platform.
>
> Thanks,
> Justin
>
> 1. https://incubator.apache.org/guides/distribution.html
> -
> To unsubscribe, e-mail: dev-unsubscr...@community.apache.org
> For additional commands, e-mail: dev-h...@community.apache.org
>
>

-- 
+48 660 796 129

Re: Migration and consolidation helm charts for ASF projects from helm/charts to apache/charts git

[Justin Mclean]

Hi,

> I am not even sure myself if I am the only one who feels the disconnection
> between reality and the current policies, that's why I started the thread
> here.

You would not be the only person who set this. People and project are making 
small steps to correct this. Recently Infra’s distribution policy was updated 
and the Incubator made these guidelines. [1] Note that Infra do support docker 
as a platform.

Thanks,
Justin

1. https://incubator.apache.org/guides/distribution.html
-
To unsubscribe, e-mail: dev-unsubscr...@community.apache.org
For additional commands, e-mail: dev-h...@community.apache.org

Re: Migration and consolidation helm charts for ASF projects from helm/charts to apache/charts git

[Matt Sicker]

Keep in mind that legal doesn’t set policy. That’s the board. Legal helps
inform the board (and whoever else) about the risks of different legal
choices. Small nitpick, but keep in mind that legal isn’t here to block us.
They’re here to help us understand the risks of which policy we choose to
enact.

On Sat, Oct 17, 2020 at 03:04 Jarek Potiuk  wrote:

> Good. So let me try to contact the legal first and try to see what they
> say. I indeed had comments that it will be difficult to get through the
> approval process, but I am eager to try this and happy to lead the effort.
> I think if I get at least a supportive voice from several projects who also
> would like to publish helm charts and images, that would be great - and
> make the case stronger. I perfectly understand that any such change is a
> serious matter and that there will be some push-back (as it is on any
> change that involves some legal consequences). I do expect quite some
> scrutiny and long process of getting there, possibly even dropping the idea
> altogether if there will be some strong case against my proposal - I am not
> a lawyer so I might be very wrong with my licence/legal understanding (and
> since ASF is a US-based company, it's not even my native Polish Law system
> that is involved here). But I am willing to try anyway.
>
> Dave says about perfectionism being an enemy. But I think it's important to
> understand both the benefits and constraints coming from belonging to ASF
> and where there are legal matters involved, I treat it very, very
> seriously.
> When I became PMC I was informed by ASF that they indemnify me from any
> legal consequences of decisions made regarding releasing the software,
> PROVIDING that I follow the policies.
>
> As we are nearing to release Airflow 2.0. Airflow is a very
> corporate/enterprise-centric software. It is used by pretty much all
> Fortune 100 companies rather heavily and for very serious stuff and
> processing really serious amounts of serious data (including a lot of
> companies that are processing it for COVID-related data crunching).
>
> I would love the ASF to both - hold their promise of indemnifying me and
> other PMC members, as well as give us the policies that allow us to operate
> efficiently, legally for our project as PMC and simply that we are not
> afraid to make decisions which might have heavy legal consequences for us.
> The current situation where the boundaries are rather blurred, is not very
> PMC-friendly. I actually spoke about it at the ApacheCon at my "Welcoming
> community strengthen the Apache Way" talk two weeks ago (I hope it will be
> released soon) that the community should be welcoming to all the members.
> Those members being users, contributors, committers, PMC, companies using
> the software, companies that are building their business on the Apache
> Software, but also -  PMCs who have legal responsibilities. "Community over
> code" is the ASF motto and I think it is important to keep it in mind.
>
> I am not even sure myself if I am the only one who feels the disconnection
> between reality and the current policies, that's why I started the thread
> here. And I am happy to invest quite some personal time to hash it out and
> work together with the Legal, Board or whoever will be involved. The more
> PMCs will at least not object or - better - understand and support me in
> that, the stronger case we can make to the Legal and Board.
>
> J.
>
>
> On Fri, Oct 16, 2020 at 9:10 PM Matthias Seidel <
> matthias.sei...@hamburg.de>
> wrote:
>
> > Really, I don't even know what they are talking about...
> > Has anyone been informed about what they are trying to achieve?
> >
> > What "significant push back from Legal and the Board" should we expect?
> >
> > Regards,
> >
> >Matthias
> >
> > Am 16.10.20 um 20:59 schrieb Joan Touzet:
> > > Hi Jarek,
> > >
> > > On 16/10/2020 05:45, Jarek Potiuk wrote:
> > >
> > >> Joan - I hope you are back and we can continue the discussion.
> > > Sorry, I just don't have the time or the energy to carry this further.
> > >
> > > I only brought up OO because it was one of the outliers, and the first
> > > "user application" that came to mind when considering Apache projects
> > > that deliver actual end user software vs. e.g. Java libraries. All
> > > deference to Dave Fisher and the OO PMC; they have many concerns that
> > > extend far beyond the terms of this policy. Apologies for pulling that
> > > community into this thread.
> > >
> > > I'm glad my comments have given you some food for thought. I would
> > > expect significant push back from Legal and the Board. That said, good
> > > luck with your efforts.
> > >
> > > Please drop me from this thread.
> > >
> > > -Joan
> > >
> > >> I'd love
> > >> to come up with the proposal that will include the specifics of
> OOffice
> > >> releases. The proposal is here -
> > >>
> >
> https://cwiki.apache.org/confluence/display/COMDEV/Updates+of+policies+for+the+convenience+packages
> > >> 

Re: Migration and consolidation helm charts for ASF projects from helm/charts to apache/charts git

[Jarek Potiuk]

Good. So let me try to contact the legal first and try to see what they
say. I indeed had comments that it will be difficult to get through the
approval process, but I am eager to try this and happy to lead the effort.
I think if I get at least a supportive voice from several projects who also
would like to publish helm charts and images, that would be great - and
make the case stronger. I perfectly understand that any such change is a
serious matter and that there will be some push-back (as it is on any
change that involves some legal consequences). I do expect quite some
scrutiny and long process of getting there, possibly even dropping the idea
altogether if there will be some strong case against my proposal - I am not
a lawyer so I might be very wrong with my licence/legal understanding (and
since ASF is a US-based company, it's not even my native Polish Law system
that is involved here). But I am willing to try anyway.

Dave says about perfectionism being an enemy. But I think it's important to
understand both the benefits and constraints coming from belonging to ASF
and where there are legal matters involved, I treat it very, very
seriously.
When I became PMC I was informed by ASF that they indemnify me from any
legal consequences of decisions made regarding releasing the software,
PROVIDING that I follow the policies.

As we are nearing to release Airflow 2.0. Airflow is a very
corporate/enterprise-centric software. It is used by pretty much all
Fortune 100 companies rather heavily and for very serious stuff and
processing really serious amounts of serious data (including a lot of
companies that are processing it for COVID-related data crunching).

I would love the ASF to both - hold their promise of indemnifying me and
other PMC members, as well as give us the policies that allow us to operate
efficiently, legally for our project as PMC and simply that we are not
afraid to make decisions which might have heavy legal consequences for us.
The current situation where the boundaries are rather blurred, is not very
PMC-friendly. I actually spoke about it at the ApacheCon at my "Welcoming
community strengthen the Apache Way" talk two weeks ago (I hope it will be
released soon) that the community should be welcoming to all the members.
Those members being users, contributors, committers, PMC, companies using
the software, companies that are building their business on the Apache
Software, but also -  PMCs who have legal responsibilities. "Community over
code" is the ASF motto and I think it is important to keep it in mind.

I am not even sure myself if I am the only one who feels the disconnection
between reality and the current policies, that's why I started the thread
here. And I am happy to invest quite some personal time to hash it out and
work together with the Legal, Board or whoever will be involved. The more
PMCs will at least not object or - better - understand and support me in
that, the stronger case we can make to the Legal and Board.

J.


On Fri, Oct 16, 2020 at 9:10 PM Matthias Seidel 
wrote:

> Really, I don't even know what they are talking about...
> Has anyone been informed about what they are trying to achieve?
>
> What "significant push back from Legal and the Board" should we expect?
>
> Regards,
>
>Matthias
>
> Am 16.10.20 um 20:59 schrieb Joan Touzet:
> > Hi Jarek,
> >
> > On 16/10/2020 05:45, Jarek Potiuk wrote:
> >
> >> Joan - I hope you are back and we can continue the discussion.
> > Sorry, I just don't have the time or the energy to carry this further.
> >
> > I only brought up OO because it was one of the outliers, and the first
> > "user application" that came to mind when considering Apache projects
> > that deliver actual end user software vs. e.g. Java libraries. All
> > deference to Dave Fisher and the OO PMC; they have many concerns that
> > extend far beyond the terms of this policy. Apologies for pulling that
> > community into this thread.
> >
> > I'm glad my comments have given you some food for thought. I would
> > expect significant push back from Legal and the Board. That said, good
> > luck with your efforts.
> >
> > Please drop me from this thread.
> >
> > -Joan
> >
> >> I'd love
> >> to come up with the proposal that will include the specifics of OOffice
> >> releases. The proposal is here -
> >>
> https://cwiki.apache.org/confluence/display/COMDEV/Updates+of+policies+for+the+convenience+packages
> >> <
> https://cwiki.apache.org/confluence/display/COMDEV/Updates+of+policies+for+the+convenience+packages
> >.
> >> Happy to hear some responses on my comments/proposal
> >> Also anyone who is interested - I invite you to comment on the proposal.
> >>
> >> My plan is to - possibly - come up with a proposal that we all
> >> people here will be ok with (hopefully next week) and submit it to the
> >> legal team of ASF for review/opinions.
> >>
> >> J.
> >>
> >>
> >> On Tue, Sep 15, 2020 at 2:06 PM Jarek Potiuk  >> > wrote:
> >>
> >> Coo

Re: Migration and consolidation helm charts for ASF projects from helm/charts to apache/charts git

[Matthias Seidel]

Really, I don't even know what they are talking about...
Has anyone been informed about what they are trying to achieve?

What "significant push back from Legal and the Board" should we expect?

Regards,

   Matthias

Am 16.10.20 um 20:59 schrieb Joan Touzet:
> Hi Jarek,
>
> On 16/10/2020 05:45, Jarek Potiuk wrote:
>
>> Joan - I hope you are back and we can continue the discussion. 
> Sorry, I just don't have the time or the energy to carry this further.
>
> I only brought up OO because it was one of the outliers, and the first
> "user application" that came to mind when considering Apache projects
> that deliver actual end user software vs. e.g. Java libraries. All
> deference to Dave Fisher and the OO PMC; they have many concerns that
> extend far beyond the terms of this policy. Apologies for pulling that
> community into this thread.
>
> I'm glad my comments have given you some food for thought. I would
> expect significant push back from Legal and the Board. That said, good
> luck with your efforts.
>
> Please drop me from this thread.
>
> -Joan
>
>> I'd love
>> to come up with the proposal that will include the specifics of OOffice
>> releases. The proposal is here -
>> https://cwiki.apache.org/confluence/display/COMDEV/Updates+of+policies+for+the+convenience+packages
>> .
>> Happy to hear some responses on my comments/proposal
>> Also anyone who is interested - I invite you to comment on the proposal.
>>
>> My plan is to - possibly - come up with a proposal that we all
>> people here will be ok with (hopefully next week) and submit it to the
>> legal team of ASF for review/opinions.
>>
>> J.
>>
>>
>> On Tue, Sep 15, 2020 at 2:06 PM Jarek Potiuk > > wrote:
>>
>> Cool. Thanks Bertrand - I am aware of some of those, but this list
>> will be helpful so I can make some of the references to those in my
>> proposal (and I encourage everyone else to do so). I am super-happy
>> to merge yours with mine. I believe the "spirit" of those is rather
>> similar. I am fully aware legal review should be done - I want now
>> to get some initial feedback and see what controversies the proposal
>> raises and polish it a bit, but I 100% understand the policies are
>> binding for the ASF and the risk and legal side of it should be very
>> carefully considered.
>>
>> Note - I just run through a few of the comments we have there and
>> just for the sake of keeping people informed on what has changed so
>> far here are some "gists" of my changes comparing to the first draft:
>>
>> * there is an open question about the viability of putting all the
>> instructions or scripts to build the binary dependencies into 
>> released sources. Giving the example of OpenOffice, CouchDB and
>> Erlang which makes it next to impossible to do. So I proposed to
>> explicitly say that any form of the instructions: scripts, manual
>> instruction or POINTERS to the right instructions is fine. Simple
>> HTTP link where you can find how to build an external OSS library
>> should be perfectly fine. My ultimate goal is that whatever whenever
>> the source .tar.gz package is created - the goal is that the user
>> can get the sources and following the instructions (including the
>> links to instructions) can - potentially rebuild the software
>> legally. It might be a complex and recursive process (build a
>> library to build a library) and at times those instructions might
>> not work (as it is with all the instructions) but at least an
>> attempt should be made to make it possible.
>>
>> * The "official" 3rd-party binary package is not a good name - I
>> replaced it for now with the "maintained OSS" binary package. The
>> idea behind it is that shortly - it should be open-source and it
>> should be maintained. So while the name does not reflect all the
>> subtleties of "maintained" and "OSS" but it reflects the spirit. I
>> tried to make the "recursive" definition as much relaxed as possible
>> (in terms of SHOULD vs. MUST except with the Licencing which is a MUST) 
>>
>> * In pretty much all cases where I write about "best practices",
>> they are not absolute requirements - so whenever possible they are
>> SHOULD instead of MUST. I am very far from imposing all the best
>> practices on all ASF projects - that will be impractical and stupid
>> thing to do. I really treat those "best practices" as "beacons" -
>> targets that we can have in mind but might never fully achieve them.
>> And as long as we have good reason, not to follow those practices -
>> by all means we do not have to. But if easy and possible, I see the
>> best practices as a powerful message that improves the "Brand" of
>> ASF in general from the user perspective. There are no "bonus
>>

Re: Migration and consolidation helm charts for ASF projects from helm/charts to apache/charts git

[Joan Touzet]

Hi Jarek,

On 16/10/2020 05:45, Jarek Potiuk wrote:

> Joan - I hope you are back and we can continue the discussion. 

Sorry, I just don't have the time or the energy to carry this further.

I only brought up OO because it was one of the outliers, and the first
"user application" that came to mind when considering Apache projects
that deliver actual end user software vs. e.g. Java libraries. All
deference to Dave Fisher and the OO PMC; they have many concerns that
extend far beyond the terms of this policy. Apologies for pulling that
community into this thread.

I'm glad my comments have given you some food for thought. I would
expect significant push back from Legal and the Board. That said, good
luck with your efforts.

Please drop me from this thread.

-Joan

> I'd love
> to come up with the proposal that will include the specifics of OOffice
> releases. The proposal is here -
> https://cwiki.apache.org/confluence/display/COMDEV/Updates+of+policies+for+the+convenience+packages
> .
> Happy to hear some responses on my comments/proposal
> Also anyone who is interested - I invite you to comment on the proposal.
> 
> My plan is to - possibly - come up with a proposal that we all
> people here will be ok with (hopefully next week) and submit it to the
> legal team of ASF for review/opinions.
> 
> J.
> 
> 
> On Tue, Sep 15, 2020 at 2:06 PM Jarek Potiuk  > wrote:
> 
> Cool. Thanks Bertrand - I am aware of some of those, but this list
> will be helpful so I can make some of the references to those in my
> proposal (and I encourage everyone else to do so). I am super-happy
> to merge yours with mine. I believe the "spirit" of those is rather
> similar. I am fully aware legal review should be done - I want now
> to get some initial feedback and see what controversies the proposal
> raises and polish it a bit, but I 100% understand the policies are
> binding for the ASF and the risk and legal side of it should be very
> carefully considered.
> 
> Note - I just run through a few of the comments we have there and
> just for the sake of keeping people informed on what has changed so
> far here are some "gists" of my changes comparing to the first draft:
> 
> * there is an open question about the viability of putting all the
> instructions or scripts to build the binary dependencies into 
> released sources. Giving the example of OpenOffice, CouchDB and
> Erlang which makes it next to impossible to do. So I proposed to
> explicitly say that any form of the instructions: scripts, manual
> instruction or POINTERS to the right instructions is fine. Simple
> HTTP link where you can find how to build an external OSS library
> should be perfectly fine. My ultimate goal is that whatever whenever
> the source .tar.gz package is created - the goal is that the user
> can get the sources and following the instructions (including the
> links to instructions) can - potentially rebuild the software
> legally. It might be a complex and recursive process (build a
> library to build a library) and at times those instructions might
> not work (as it is with all the instructions) but at least an
> attempt should be made to make it possible.
> 
> * The "official" 3rd-party binary package is not a good name - I
> replaced it for now with the "maintained OSS" binary package. The
> idea behind it is that shortly - it should be open-source and it
> should be maintained. So while the name does not reflect all the
> subtleties of "maintained" and "OSS" but it reflects the spirit. I
> tried to make the "recursive" definition as much relaxed as possible
> (in terms of SHOULD vs. MUST except with the Licencing which is a MUST) 
> 
> * In pretty much all cases where I write about "best practices",
> they are not absolute requirements - so whenever possible they are
> SHOULD instead of MUST. I am very far from imposing all the best
> practices on all ASF projects - that will be impractical and stupid
> thing to do. I really treat those "best practices" as "beacons" -
> targets that we can have in mind but might never fully achieve them.
> And as long as we have good reason, not to follow those practices -
> by all means we do not have to. But if easy and possible, I see the
> best practices as a powerful message that improves the "Brand" of
> ASF in general from the user perspective. There are no "bonus
> points" for projects that follow it vs. those which decided not to
> in particular cases. But having those as "targets" for ASF projects
> is an important message.
> 
> J,
> 
> 
> 
> On Tue, Sep 15, 2020 at 12:25 PM Bertrand Delacretaz
> mailto:bdelacre...@apache.org>> wrote:
> 
> Hi,
> 
> On Mon, Sep 7, 2020 a

Re: Migration and consolidation helm charts for ASF projects from helm/charts to apache/charts git

[Dave Fisher]

Let’s keep this simple.

Focus on your needs for clarification for Helm Charts.

I really don’t have the time or inclination to parse out what troubles there 
are for OpenOffice in your plan. I already have a long list with unexpected 
issues and mandates.

The Foundation exists for its Project Communities. The Board will need to 
decide if it will mandate any changes to PMCs.

VP, Legal Affairs has been delegated the responsibility for Release Policies.

Mandating changes to volunteers is like pushing a string and herding cats.

I don’t mean to be harsh and do want your effort to be successful. Don’t let 
perfection be your enemy.

Best Regards,
Dave

Sent from my iPhone

> On Oct 16, 2020, at 7:58 AM, Jarek Potiuk  wrote:
> 
> I do not want to make any assumptions on Windows/MacOS etc - for me this is
> really just proposal to clarify the ASF policies which (as far as at
> least I understand) are not up-to-date with some distribution mechanisms
> for convenience packages that have become popular way that our users can
> conveniently use to install our software.
> 
> The OpenOffice came because Joan stepped up and raised the point that
> current proposal is not ok for OOofice so I tried to come up with ideas how
> to make it works also for OOffice. I hope discussing it here at the devcom
> is the right place to do.
> 
> Would you be the right person to comment on the proposal Dave from the Open
> Office team? I see you are on the PMC list of Open Office? Would you be
> happy with the proposal? Or maybe you think it is not needed ?
> 
> Just to clarify my intentions (for the whole proposal):
> 
> I just want to make sure that those who are interested in clarifying the
> policy in the way that will make it easier for PMCs to follow the policies
> and make sure that those convenience packages are prepared accordingly,
> including the indemnification promise that ASF gives to the PMCs.
> According the the PMC gude: http://www.apache.org/dev/pmc.html it's the
> PMCs responsibility to follow the policies:
> 
> COMPLY WITH LEGAL AFFAIRS POLICIES¶
> PMCs SHALL ensure that the work on their project and the code that they
> produce comply with relevant Legal Affairs Committee policies, including
> appropriately using the Apache License, handling IP and copyrights
> correctly, handling cryptography, and producing official software releases
> of their products.
> 
> As a PMC of Apache Airflow I simply find it difficult to follow when it
> comes to helm charts/docker images, that's why I came forward with the
> proposal how to modify those policies to adapt to those distribution
> mechanisms - and my intention is to address any concerns other PMC might
> have here. I hope we can collaborate on that.
> 
> J.
> 
> 
>> On Fri, Oct 16, 2020 at 3:05 PM Dave Fisher  wrote:
>> 
>> Questions related to OpenOffice must be discussed with the OpenOffice PMC.
>> Joan is an observer and I don’t recall her asking the PMC.
>> 
>> Please don’t make any assumptions about builds for Windows, macOS (we
>> support 10.7 and newer) and Linux. Community members on their own build
>> OS/2 and FreeBSD versions.
>> 
>> OpenOffice is completely volunteer driven.
>> 
>> Many in the community are not native English speakers.
>> 
>> Tread carefully.
>> 
>> Regards,
>> Dave
>> 
>> Sent from my iPhone
>> 
 On Oct 16, 2020, at 2:46 AM, Jarek Potiuk  wrote:
>>> 
>>> Hello everyone,
>>> 
>>> As Apache Airflow 2.0 alpha is out, I have some time to come back to the
>>> discussion. I also add Joan who was discussing the policy in a
>>> separate thread in the builds@ devlist:
>>> 
>> https://lists.apache.org/thread.html/r18e21d4719755f83fc67a81c842f7a59a7aa76cd1f621b1362cdd6ae%40%3Cbuilds.apache.org%3E
>>> 
>>> Joan - I hope you are back and we can continue the discussion. I'd love
>> to
>>> come up with the proposal that will include the specifics of OOffice
>>> releases. The proposal is here -
>>> 
>> https://cwiki.apache.org/confluence/display/COMDEV/Updates+of+policies+for+the+convenience+packages
>> .
>>> Happy to hear some responses on my comments/proposal
>>> Also anyone who is interested - I invite you to comment on the proposal.
>>> 
>>> My plan is to - possibly - come up with a proposal that we all people
>> here
>>> will be ok with (hopefully next week) and submit it to the legal team of
>>> ASF for review/opinions.
>>> 
>>> J.
>>> 
>>> 
 On Tue, Sep 15, 2020 at 2:06 PM Jarek Potiuk  wrote:
 
 Cool. Thanks Bertrand - I am aware of some of those, but this list will
>> be
 helpful so I can make some of the references to those in my proposal
>> (and I
 encourage everyone else to do so). I am super-happy to merge yours with
 mine. I believe the "spirit" of those is rather similar. I am fully
>> aware
 legal review should be done - I want now to get some initial feedback
>> and
 see what controversies the proposal raises and polish it a bit, but I
>> 100%
 understand the policies are binding for the ASF and t

Re: Migration and consolidation helm charts for ASF projects from helm/charts to apache/charts git

[Jarek Potiuk]

I do not want to make any assumptions on Windows/MacOS etc - for me this is
really just proposal to clarify the ASF policies which (as far as at
least I understand) are not up-to-date with some distribution mechanisms
for convenience packages that have become popular way that our users can
conveniently use to install our software.

The OpenOffice came because Joan stepped up and raised the point that
current proposal is not ok for OOofice so I tried to come up with ideas how
to make it works also for OOffice. I hope discussing it here at the devcom
is the right place to do.

Would you be the right person to comment on the proposal Dave from the Open
Office team? I see you are on the PMC list of Open Office? Would you be
happy with the proposal? Or maybe you think it is not needed ?

Just to clarify my intentions (for the whole proposal):

I just want to make sure that those who are interested in clarifying the
policy in the way that will make it easier for PMCs to follow the policies
and make sure that those convenience packages are prepared accordingly,
including the indemnification promise that ASF gives to the PMCs.
According the the PMC gude: http://www.apache.org/dev/pmc.html it's the
PMCs responsibility to follow the policies:

COMPLY WITH LEGAL AFFAIRS POLICIES¶
PMCs SHALL ensure that the work on their project and the code that they
produce comply with relevant Legal Affairs Committee policies, including
appropriately using the Apache License, handling IP and copyrights
correctly, handling cryptography, and producing official software releases
of their products.

As a PMC of Apache Airflow I simply find it difficult to follow when it
comes to helm charts/docker images, that's why I came forward with the
proposal how to modify those policies to adapt to those distribution
mechanisms - and my intention is to address any concerns other PMC might
have here. I hope we can collaborate on that.

J.


On Fri, Oct 16, 2020 at 3:05 PM Dave Fisher  wrote:

> Questions related to OpenOffice must be discussed with the OpenOffice PMC.
> Joan is an observer and I don’t recall her asking the PMC.
>
> Please don’t make any assumptions about builds for Windows, macOS (we
> support 10.7 and newer) and Linux. Community members on their own build
> OS/2 and FreeBSD versions.
>
> OpenOffice is completely volunteer driven.
>
> Many in the community are not native English speakers.
>
> Tread carefully.
>
> Regards,
> Dave
>
> Sent from my iPhone
>
> > On Oct 16, 2020, at 2:46 AM, Jarek Potiuk  wrote:
> >
> > Hello everyone,
> >
> > As Apache Airflow 2.0 alpha is out, I have some time to come back to the
> > discussion. I also add Joan who was discussing the policy in a
> > separate thread in the builds@ devlist:
> >
> https://lists.apache.org/thread.html/r18e21d4719755f83fc67a81c842f7a59a7aa76cd1f621b1362cdd6ae%40%3Cbuilds.apache.org%3E
> >
> > Joan - I hope you are back and we can continue the discussion. I'd love
> to
> > come up with the proposal that will include the specifics of OOffice
> > releases. The proposal is here -
> >
> https://cwiki.apache.org/confluence/display/COMDEV/Updates+of+policies+for+the+convenience+packages
> .
> > Happy to hear some responses on my comments/proposal
> > Also anyone who is interested - I invite you to comment on the proposal.
> >
> > My plan is to - possibly - come up with a proposal that we all people
> here
> > will be ok with (hopefully next week) and submit it to the legal team of
> > ASF for review/opinions.
> >
> > J.
> >
> >
> >> On Tue, Sep 15, 2020 at 2:06 PM Jarek Potiuk  wrote:
> >>
> >> Cool. Thanks Bertrand - I am aware of some of those, but this list will
> be
> >> helpful so I can make some of the references to those in my proposal
> (and I
> >> encourage everyone else to do so). I am super-happy to merge yours with
> >> mine. I believe the "spirit" of those is rather similar. I am fully
> aware
> >> legal review should be done - I want now to get some initial feedback
> and
> >> see what controversies the proposal raises and polish it a bit, but I
> 100%
> >> understand the policies are binding for the ASF and the risk and legal
> side
> >> of it should be very carefully considered.
> >>
> >> Note - I just run through a few of the comments we have there and just
> for
> >> the sake of keeping people informed on what has changed so far here are
> >> some "gists" of my changes comparing to the first draft:
> >>
> >> * there is an open question about the viability of putting all the
> >> instructions or scripts to build the binary dependencies into  released
> >> sources. Giving the example of OpenOffice, CouchDB and Erlang which
> makes
> >> it next to impossible to do. So I proposed to explicitly say that any
> form
> >> of the instructions: scripts, manual instruction or POINTERS to the
> right
> >> instructions is fine. Simple HTTP link where you can find how to build
> an
> >> external OSS library should be perfectly fine. My ultimate goal is that
> >> whatever w

Re: Migration and consolidation helm charts for ASF projects from helm/charts to apache/charts git

[Dave Fisher]

Questions related to OpenOffice must be discussed with the OpenOffice PMC. Joan 
is an observer and I don’t recall her asking the PMC.

Please don’t make any assumptions about builds for Windows, macOS (we support 
10.7 and newer) and Linux. Community members on their own build OS/2 and 
FreeBSD versions.

OpenOffice is completely volunteer driven. 

Many in the community are not native English speakers.

Tread carefully.

Regards,
Dave

Sent from my iPhone

> On Oct 16, 2020, at 2:46 AM, Jarek Potiuk  wrote:
> 
> Hello everyone,
> 
> As Apache Airflow 2.0 alpha is out, I have some time to come back to the
> discussion. I also add Joan who was discussing the policy in a
> separate thread in the builds@ devlist:
> https://lists.apache.org/thread.html/r18e21d4719755f83fc67a81c842f7a59a7aa76cd1f621b1362cdd6ae%40%3Cbuilds.apache.org%3E
> 
> Joan - I hope you are back and we can continue the discussion. I'd love to
> come up with the proposal that will include the specifics of OOffice
> releases. The proposal is here -
> https://cwiki.apache.org/confluence/display/COMDEV/Updates+of+policies+for+the+convenience+packages.
> Happy to hear some responses on my comments/proposal
> Also anyone who is interested - I invite you to comment on the proposal.
> 
> My plan is to - possibly - come up with a proposal that we all people here
> will be ok with (hopefully next week) and submit it to the legal team of
> ASF for review/opinions.
> 
> J.
> 
> 
>> On Tue, Sep 15, 2020 at 2:06 PM Jarek Potiuk  wrote:
>> 
>> Cool. Thanks Bertrand - I am aware of some of those, but this list will be
>> helpful so I can make some of the references to those in my proposal (and I
>> encourage everyone else to do so). I am super-happy to merge yours with
>> mine. I believe the "spirit" of those is rather similar. I am fully aware
>> legal review should be done - I want now to get some initial feedback and
>> see what controversies the proposal raises and polish it a bit, but I 100%
>> understand the policies are binding for the ASF and the risk and legal side
>> of it should be very carefully considered.
>> 
>> Note - I just run through a few of the comments we have there and just for
>> the sake of keeping people informed on what has changed so far here are
>> some "gists" of my changes comparing to the first draft:
>> 
>> * there is an open question about the viability of putting all the
>> instructions or scripts to build the binary dependencies into  released
>> sources. Giving the example of OpenOffice, CouchDB and Erlang which makes
>> it next to impossible to do. So I proposed to explicitly say that any form
>> of the instructions: scripts, manual instruction or POINTERS to the right
>> instructions is fine. Simple HTTP link where you can find how to build an
>> external OSS library should be perfectly fine. My ultimate goal is that
>> whatever whenever the source .tar.gz package is created - the goal is that
>> the user can get the sources and following the instructions (including the
>> links to instructions) can - potentially rebuild the software legally. It
>> might be a complex and recursive process (build a library to build a
>> library) and at times those instructions might not work (as it is with all
>> the instructions) but at least an attempt should be made to make it
>> possible.
>> 
>> * The "official" 3rd-party binary package is not a good name - I replaced
>> it for now with the "maintained OSS" binary package. The idea behind it is
>> that shortly - it should be open-source and it should be maintained. So
>> while the name does not reflect all the subtleties of "maintained" and
>> "OSS" but it reflects the spirit. I tried to make the "recursive"
>> definition as much relaxed as possible (in terms of SHOULD vs. MUST except
>> with the Licencing which is a MUST)
>> 
>> * In pretty much all cases where I write about "best practices", they are
>> not absolute requirements - so whenever possible they are SHOULD instead of
>> MUST. I am very far from imposing all the best practices on all ASF
>> projects - that will be impractical and stupid thing to do. I really treat
>> those "best practices" as "beacons" - targets that we can have in mind but
>> might never fully achieve them. And as long as we have good reason, not to
>> follow those practices - by all means we do not have to. But if easy and
>> possible, I see the best practices as a powerful message that improves the
>> "Brand" of ASF in general from the user perspective. There are no "bonus
>> points" for projects that follow it vs. those which decided not to in
>> particular cases. But having those as "targets" for ASF projects is an
>> important message.
>> 
>> J,
>> 
>> 
>> 
>> On Tue, Sep 15, 2020 at 12:25 PM Bertrand Delacretaz <
>> bdelacre...@apache.org> wrote:
>> 
>>> Hi,
>>> 
>>> On Mon, Sep 7, 2020 at 10:24 AM Bertrand Delacretaz
>>>  wrote:
>>> ...
 ...* https://issues.apache.org/jira/browse/ASFP-23 (accessible to ASF
 Members) has links to relate

Re: Migration and consolidation helm charts for ASF projects from helm/charts to apache/charts git

[Jarek Potiuk]

Hello everyone,

As Apache Airflow 2.0 alpha is out, I have some time to come back to the
discussion. I also add Joan who was discussing the policy in a
separate thread in the builds@ devlist:
https://lists.apache.org/thread.html/r18e21d4719755f83fc67a81c842f7a59a7aa76cd1f621b1362cdd6ae%40%3Cbuilds.apache.org%3E

Joan - I hope you are back and we can continue the discussion. I'd love to
come up with the proposal that will include the specifics of OOffice
releases. The proposal is here -
https://cwiki.apache.org/confluence/display/COMDEV/Updates+of+policies+for+the+convenience+packages.
Happy to hear some responses on my comments/proposal
Also anyone who is interested - I invite you to comment on the proposal.

My plan is to - possibly - come up with a proposal that we all people here
will be ok with (hopefully next week) and submit it to the legal team of
ASF for review/opinions.

J.


On Tue, Sep 15, 2020 at 2:06 PM Jarek Potiuk  wrote:

> Cool. Thanks Bertrand - I am aware of some of those, but this list will be
> helpful so I can make some of the references to those in my proposal (and I
> encourage everyone else to do so). I am super-happy to merge yours with
> mine. I believe the "spirit" of those is rather similar. I am fully aware
> legal review should be done - I want now to get some initial feedback and
> see what controversies the proposal raises and polish it a bit, but I 100%
> understand the policies are binding for the ASF and the risk and legal side
> of it should be very carefully considered.
>
> Note - I just run through a few of the comments we have there and just for
> the sake of keeping people informed on what has changed so far here are
> some "gists" of my changes comparing to the first draft:
>
> * there is an open question about the viability of putting all the
> instructions or scripts to build the binary dependencies into  released
> sources. Giving the example of OpenOffice, CouchDB and Erlang which makes
> it next to impossible to do. So I proposed to explicitly say that any form
> of the instructions: scripts, manual instruction or POINTERS to the right
> instructions is fine. Simple HTTP link where you can find how to build an
> external OSS library should be perfectly fine. My ultimate goal is that
> whatever whenever the source .tar.gz package is created - the goal is that
> the user can get the sources and following the instructions (including the
> links to instructions) can - potentially rebuild the software legally. It
> might be a complex and recursive process (build a library to build a
> library) and at times those instructions might not work (as it is with all
> the instructions) but at least an attempt should be made to make it
> possible.
>
> * The "official" 3rd-party binary package is not a good name - I replaced
> it for now with the "maintained OSS" binary package. The idea behind it is
> that shortly - it should be open-source and it should be maintained. So
> while the name does not reflect all the subtleties of "maintained" and
> "OSS" but it reflects the spirit. I tried to make the "recursive"
> definition as much relaxed as possible (in terms of SHOULD vs. MUST except
> with the Licencing which is a MUST)
>
> * In pretty much all cases where I write about "best practices", they are
> not absolute requirements - so whenever possible they are SHOULD instead of
> MUST. I am very far from imposing all the best practices on all ASF
> projects - that will be impractical and stupid thing to do. I really treat
> those "best practices" as "beacons" - targets that we can have in mind but
> might never fully achieve them. And as long as we have good reason, not to
> follow those practices - by all means we do not have to. But if easy and
> possible, I see the best practices as a powerful message that improves the
> "Brand" of ASF in general from the user perspective. There are no "bonus
> points" for projects that follow it vs. those which decided not to in
> particular cases. But having those as "targets" for ASF projects is an
> important message.
>
> J,
>
>
>
> On Tue, Sep 15, 2020 at 12:25 PM Bertrand Delacretaz <
> bdelacre...@apache.org> wrote:
>
>> Hi,
>>
>> On Mon, Sep 7, 2020 at 10:24 AM Bertrand Delacretaz
>>  wrote:
>> ...
>> > ...* https://issues.apache.org/jira/browse/ASFP-23 (accessible to ASF
>> > Members) has links to related discussions...
>>
>> For the benefit of people who don't have access to that ticket, here's
>> a list of links that are public and can help inform this discussion.
>>
>> -
>> https://cwiki.apache.org/confluence/display/INCUBATOR/Distribution+Guidelines
>> mentions Maven, GitHub, Docker and other similar distribution
>> channels. The topic was discussed several times in the Incubator,
>> including
>>
>> https://lists.apache.org/thread.html/f614a20f82b010d152ce3871165841810c0db6c4de9d8a27e78b71d6@%3Clegal-discuss.apache.org%3E
>>
>> - LEGAL- tickets, https://issues.apache.org/jira/browse/LEGAL-270 ,
>> https://issues.apache.org/ji

Re: Migration and consolidation helm charts for ASF projects from helm/charts to apache/charts git

[Jarek Potiuk]

Cool. Thanks Bertrand - I am aware of some of those, but this list will be
helpful so I can make some of the references to those in my proposal (and I
encourage everyone else to do so). I am super-happy to merge yours with
mine. I believe the "spirit" of those is rather similar. I am fully aware
legal review should be done - I want now to get some initial feedback and
see what controversies the proposal raises and polish it a bit, but I 100%
understand the policies are binding for the ASF and the risk and legal side
of it should be very carefully considered.

Note - I just run through a few of the comments we have there and just for
the sake of keeping people informed on what has changed so far here are
some "gists" of my changes comparing to the first draft:

* there is an open question about the viability of putting all the
instructions or scripts to build the binary dependencies into  released
sources. Giving the example of OpenOffice, CouchDB and Erlang which makes
it next to impossible to do. So I proposed to explicitly say that any form
of the instructions: scripts, manual instruction or POINTERS to the right
instructions is fine. Simple HTTP link where you can find how to build an
external OSS library should be perfectly fine. My ultimate goal is that
whatever whenever the source .tar.gz package is created - the goal is that
the user can get the sources and following the instructions (including the
links to instructions) can - potentially rebuild the software legally. It
might be a complex and recursive process (build a library to build a
library) and at times those instructions might not work (as it is with all
the instructions) but at least an attempt should be made to make it
possible.

* The "official" 3rd-party binary package is not a good name - I replaced
it for now with the "maintained OSS" binary package. The idea behind it is
that shortly - it should be open-source and it should be maintained. So
while the name does not reflect all the subtleties of "maintained" and
"OSS" but it reflects the spirit. I tried to make the "recursive"
definition as much relaxed as possible (in terms of SHOULD vs. MUST except
with the Licencing which is a MUST)

* In pretty much all cases where I write about "best practices", they are
not absolute requirements - so whenever possible they are SHOULD instead of
MUST. I am very far from imposing all the best practices on all ASF
projects - that will be impractical and stupid thing to do. I really treat
those "best practices" as "beacons" - targets that we can have in mind but
might never fully achieve them. And as long as we have good reason, not to
follow those practices - by all means we do not have to. But if easy and
possible, I see the best practices as a powerful message that improves the
"Brand" of ASF in general from the user perspective. There are no "bonus
points" for projects that follow it vs. those which decided not to in
particular cases. But having those as "targets" for ASF projects is an
important message.

J,



On Tue, Sep 15, 2020 at 12:25 PM Bertrand Delacretaz 
wrote:

> Hi,
>
> On Mon, Sep 7, 2020 at 10:24 AM Bertrand Delacretaz
>  wrote:
> ...
> > ...* https://issues.apache.org/jira/browse/ASFP-23 (accessible to ASF
> > Members) has links to related discussions...
>
> For the benefit of people who don't have access to that ticket, here's
> a list of links that are public and can help inform this discussion.
>
> -
> https://cwiki.apache.org/confluence/display/INCUBATOR/Distribution+Guidelines
> mentions Maven, GitHub, Docker and other similar distribution
> channels. The topic was discussed several times in the Incubator,
> including
>
> https://lists.apache.org/thread.html/f614a20f82b010d152ce3871165841810c0db6c4de9d8a27e78b71d6@%3Clegal-discuss.apache.org%3E
>
> - LEGAL- tickets, https://issues.apache.org/jira/browse/LEGAL-270 ,
> https://issues.apache.org/jira/browse/LEGAL-427 ,
> https://issues.apache.org/jira/browse/LEGAL-323 (linked to a number of
> others), more?
>
> - Reproducible builds (https://reproducible-builds.org/ ,
> http://maven.apache.org/guides/mini/guide-reproducible-builds.html )
> are also a way to improve trust in binaries.
>
> I also made a proposal a while ago about how we might handle these things:
>
> *** DRAFT TENTATIVE PROPOSAL ON HOW WE MIGHT EXPLAIN DISTRIBUTIONS 
> -ASF Releases consist of source code only
>
> -As a convenience, our PMCs often distribute binary packages along
> with these releases, as attachments which are not considered part of
> the release
>
> -We recommend that people build their own binaries from released code,
> but it's a reality that many of them use the binaries that we
> distribute
>
> -The only things we state about these binaries are that the PMC which
> creates them believes they are the correct ones (with no guarantees)
> and that the digests that we distribute are correct
>
> -Those digests are mentioned in the PMC approval votes for these
> binaries, to allow people to look

Re: Migration and consolidation helm charts for ASF projects from helm/charts to apache/charts git

[Bertrand Delacretaz]

Hi,

On Mon, Sep 7, 2020 at 10:24 AM Bertrand Delacretaz
 wrote:
...
> ...* https://issues.apache.org/jira/browse/ASFP-23 (accessible to ASF
> Members) has links to related discussions...

For the benefit of people who don't have access to that ticket, here's
a list of links that are public and can help inform this discussion.

- https://cwiki.apache.org/confluence/display/INCUBATOR/Distribution+Guidelines
mentions Maven, GitHub, Docker and other similar distribution
channels. The topic was discussed several times in the Incubator,
including
https://lists.apache.org/thread.html/f614a20f82b010d152ce3871165841810c0db6c4de9d8a27e78b71d6@%3Clegal-discuss.apache.org%3E

- LEGAL- tickets, https://issues.apache.org/jira/browse/LEGAL-270 ,
https://issues.apache.org/jira/browse/LEGAL-427 ,
https://issues.apache.org/jira/browse/LEGAL-323 (linked to a number of
others), more?

- Reproducible builds (https://reproducible-builds.org/ ,
http://maven.apache.org/guides/mini/guide-reproducible-builds.html )
are also a way to improve trust in binaries.

I also made a proposal a while ago about how we might handle these things:

*** DRAFT TENTATIVE PROPOSAL ON HOW WE MIGHT EXPLAIN DISTRIBUTIONS 
-ASF Releases consist of source code only

-As a convenience, our PMCs often distribute binary packages along
with these releases, as attachments which are not considered part of
the release

-We recommend that people build their own binaries from released code,
but it's a reality that many of them use the binaries that we
distribute

-The only things we state about these binaries are that the PMC which
creates them believes they are the correct ones (with no guarantees)
and that the digests that we distribute are correct

-Those digests are mentioned in the PMC approval votes for these
binaries, to allow people to look them up if desired

-We strongly encourage our PMCs to produce reproducible builds as per
https://reproducible-builds.org/
*** DRAFT TENTATIVE PROPOSAL ON HOW WE MIGHT EXPLAIN DISTRIBUTIONS 

I haven't found time to pursue it so far, and it might not be
implementable as is but hopefully it helps this discussion.

If a draft policy is created based on that and/or Jarek's current
proposal, legal review will be needed before the ASF can activate it.

-Bertrand

-
To unsubscribe, e-mail: dev-unsubscr...@community.apache.org
For additional commands, e-mail: dev-h...@community.apache.org

Re: Migration and consolidation helm charts for ASF projects from helm/charts to apache/charts git

[Jarek Potiuk]

Hello Everyone.

As promised, I prepared a draft of the proposal of changes that I would
love if a number of interested people discuss it, comment, criticise, agree
on eventually, and submit to the ASF Board for Approval.

I tried to capture all the context, but also I marked clearly all the
proposals that I think should be included in the ASF policies and clearly
marked changes that should be applied. I also tried to write it in the
"future-proof" way - I tried to make statements that do not refer to the
Images or Helm Charts, but describe general practices of "packaged"
software as opposed to "compiled" software that seems to be the origin of
the current policies. So my approach was really to try to describe and set
policies around "software packaging" in general, rather than "Images/Helm
Charts" in particular. However I believe it is much more to take the
proposed policies and apply them directly to the Images and Helm Charts
rather than the original policies.

As promised I also commented (with inline comments), the places where I
know there are some controversies - at least those that came up in our
original discussions in Airflow - and I explained how I understand the
controversies that are around that.

I would really love to get a lot of comments and discussion around the
proposal, before we submit the proposal - I am looking forward to your
comments!

The proposal is here:
https://cwiki.apache.org/confluence/display/COMDEV/Updates+of+policies+for+the+convenience+packages


J.



On Wed, Sep 9, 2020 at 10:03 PM Jarek Potiuk  wrote:

> Issue in COMDEV created: https://issues.apache.org/jira/browse/COMDEV-382
>
> I will prepare an early draft of the proposal shortly and ping here to
> spark a discussion
>
> J.
>
> On Tue, Sep 8, 2020 at 5:59 PM Matt Sicker  wrote:
>
>> I've spent an inordinate amount of time at $dayjob triaging security
>> vulnerabilities from Docker scans, so I can definitely attest to
>> Mark's experience there. In fact, one of the biggest offenders was the
>> official Docker Hub image for openjdk! Then there were a few years
>> where people pushed Alpine Linux in containers, but then maintenance
>> stopped related to that in openjdk, further leading to more outdated
>> images. Then there's the fairly broad lack of security awareness in
>> most published Docker images (e.g., running everything as root,
>> installing unnecessary dependencies, leaving behind private keys,
>> etc.) On the other hand, publishing updated images puts us back into
>> the problem of distributing non-AL software.
>>
>> Note that you could be releasing a new image every day, yet that
>> doesn't fix the problem of outdated upstream layers, nor is it easily
>> fixable by adding an "apt-get update && apt-get upgrade -y" step as
>> that breaches back into licensing complexity (Apache isn't a Linux
>> distribution after all!).
>>
>> On Tue, 8 Sep 2020 at 07:37, Jarek Potiuk  wrote:
>> >
>> > Will definitely include that in my proposal Mark!
>> >
>> > BTW. Speaking of the report you got, we got the user talking to us on
>> > slack, and got the user to retest it on the refreshed image.
>> >
>> > It all boiled down to 4 "undefined" risk issues reported by the tool
>> (seems
>> > that their - reasonable - approach is that anything High or Undefined
>> is a
>> > blocker). And the root cause in this case is that the base image that we
>> > used (python-debian-buster) contains those vulnerabilities. Most have
>> been
>> > fixed in other releases of Debian (stretch, bullseye), but the current
>> > (buster) LTS patch has been released over a month ago (3rd of August).
>> With
>> > their release cadence (~ 1/month) , we should get it sooner rather than
>> > later. And following our tooling - we will regenerate and rebuild our
>> > images once this is available (we are planning to automate this part
>> soon).
>> > And I hope most or all of those will be addressed.
>> >
>> > Following your analogy, that's indeed very true that the images age like
>> > milk, so it looks like you are supposed to replace it with a fresh
>> bottle
>> > from time to time. I will try to capture that as best practice.
>> >
>> > I am tempted to put your analogy to the proposal ;) I wonder whether the
>> > Board shares the same sense of humour.
>> >
>> > J.
>> >
>> >
>> >
>> >
>> > On Tue, Sep 8, 2020 at 12:36 PM Mark J Cox  wrote:
>> >
>> > > On Mon, Sep 7, 2020 at 2:21 PM Jarek Potiuk  wrote:
>> > >
>> > > > I also talked to the Apache Security team today (there was an issue
>> > > raised
>> > > > about the security of the images which I think should be part of the
>> > > policy
>> > > > as well.
>> > > >
>> > >
>> > > Thanks Jarek.  What happened is that we got a report to
>> > > secur...@apache.org
>> > > about a docker container that when scanned showed a lot of "unfixed
>> > > vulnerabilities". I'm using quotes there because our usual response to
>> > > people sending us unfiltered reports from scanning tools is to reject
>> them;
>> > > we get

Re: Migration and consolidation helm charts for ASF projects from helm/charts to apache/charts git

[Jarek Potiuk]

Issue in COMDEV created: https://issues.apache.org/jira/browse/COMDEV-382

I will prepare an early draft of the proposal shortly and ping here to
spark a discussion

J.

On Tue, Sep 8, 2020 at 5:59 PM Matt Sicker  wrote:

> I've spent an inordinate amount of time at $dayjob triaging security
> vulnerabilities from Docker scans, so I can definitely attest to
> Mark's experience there. In fact, one of the biggest offenders was the
> official Docker Hub image for openjdk! Then there were a few years
> where people pushed Alpine Linux in containers, but then maintenance
> stopped related to that in openjdk, further leading to more outdated
> images. Then there's the fairly broad lack of security awareness in
> most published Docker images (e.g., running everything as root,
> installing unnecessary dependencies, leaving behind private keys,
> etc.) On the other hand, publishing updated images puts us back into
> the problem of distributing non-AL software.
>
> Note that you could be releasing a new image every day, yet that
> doesn't fix the problem of outdated upstream layers, nor is it easily
> fixable by adding an "apt-get update && apt-get upgrade -y" step as
> that breaches back into licensing complexity (Apache isn't a Linux
> distribution after all!).
>
> On Tue, 8 Sep 2020 at 07:37, Jarek Potiuk  wrote:
> >
> > Will definitely include that in my proposal Mark!
> >
> > BTW. Speaking of the report you got, we got the user talking to us on
> > slack, and got the user to retest it on the refreshed image.
> >
> > It all boiled down to 4 "undefined" risk issues reported by the tool
> (seems
> > that their - reasonable - approach is that anything High or Undefined is
> a
> > blocker). And the root cause in this case is that the base image that we
> > used (python-debian-buster) contains those vulnerabilities. Most have
> been
> > fixed in other releases of Debian (stretch, bullseye), but the current
> > (buster) LTS patch has been released over a month ago (3rd of August).
> With
> > their release cadence (~ 1/month) , we should get it sooner rather than
> > later. And following our tooling - we will regenerate and rebuild our
> > images once this is available (we are planning to automate this part
> soon).
> > And I hope most or all of those will be addressed.
> >
> > Following your analogy, that's indeed very true that the images age like
> > milk, so it looks like you are supposed to replace it with a fresh bottle
> > from time to time. I will try to capture that as best practice.
> >
> > I am tempted to put your analogy to the proposal ;) I wonder whether the
> > Board shares the same sense of humour.
> >
> > J.
> >
> >
> >
> >
> > On Tue, Sep 8, 2020 at 12:36 PM Mark J Cox  wrote:
> >
> > > On Mon, Sep 7, 2020 at 2:21 PM Jarek Potiuk  wrote:
> > >
> > > > I also talked to the Apache Security team today (there was an issue
> > > raised
> > > > about the security of the images which I think should be part of the
> > > policy
> > > > as well.
> > > >
> > >
> > > Thanks Jarek.  What happened is that we got a report to
> > > secur...@apache.org
> > > about a docker container that when scanned showed a lot of "unfixed
> > > vulnerabilities". I'm using quotes there because our usual response to
> > > people sending us unfiltered reports from scanning tools is to reject
> them;
> > > we get them quite often outside of containers and binary
> distributions, and
> > > they very rarely are useful.  It's also fairly likely that the
> majority of
> > > the reported issues in the container are completely irrelevant too.
> For
> > > example the list contained a CVE for a power9 gcc issue.  These
> scanners
> > > are basically going to just report on all the things in the underlying
> base
> > > image that are not updated, and even if you recreated images every day
> > > you'd still have unfixed CVEs on the list.
> > >
> > > Containers and other similar non-source distributions don't age well (a
> > > colleague used to say they 'age like milk, not wine'), they'll collect
> more
> > > and more of these layer vulnerabilities over time, and although most
> will
> > > be irrelevant, there are going to be times when such a vulnerability
> does
> > > actually matter, and we need to make sure projects producing them have
> a
> > > process for tracking that either my monitoring (lots of effort) or by
> at
> > > least frequent rebasing to keep them fresh.
> > >
> > > That's all assuming projects are making good security decisions to
> start
> > > with; basing on images that are maintained, in life, and updated,
> making
> > > sure users know the state/freshness of them, making sure users realise
> > > there will be vulns in the underlying layers and how to escalate
> reporting
> > > vulns they find that actually are exposed to the project.  That should
> all
> > > be part of some guidelines on images.
> > >
> > > Cheers, Mark
> > > ASF Security Team
> > >
> >
> >
> > --
> > +48 660 796 129
>
>
>
> --
> Matt Sicker 
>
> --

Re: Migration and consolidation helm charts for ASF projects from helm/charts to apache/charts git

[Matt Sicker]

I've spent an inordinate amount of time at $dayjob triaging security
vulnerabilities from Docker scans, so I can definitely attest to
Mark's experience there. In fact, one of the biggest offenders was the
official Docker Hub image for openjdk! Then there were a few years
where people pushed Alpine Linux in containers, but then maintenance
stopped related to that in openjdk, further leading to more outdated
images. Then there's the fairly broad lack of security awareness in
most published Docker images (e.g., running everything as root,
installing unnecessary dependencies, leaving behind private keys,
etc.) On the other hand, publishing updated images puts us back into
the problem of distributing non-AL software.

Note that you could be releasing a new image every day, yet that
doesn't fix the problem of outdated upstream layers, nor is it easily
fixable by adding an "apt-get update && apt-get upgrade -y" step as
that breaches back into licensing complexity (Apache isn't a Linux
distribution after all!).

On Tue, 8 Sep 2020 at 07:37, Jarek Potiuk  wrote:
>
> Will definitely include that in my proposal Mark!
>
> BTW. Speaking of the report you got, we got the user talking to us on
> slack, and got the user to retest it on the refreshed image.
>
> It all boiled down to 4 "undefined" risk issues reported by the tool (seems
> that their - reasonable - approach is that anything High or Undefined is a
> blocker). And the root cause in this case is that the base image that we
> used (python-debian-buster) contains those vulnerabilities. Most have been
> fixed in other releases of Debian (stretch, bullseye), but the current
> (buster) LTS patch has been released over a month ago (3rd of August). With
> their release cadence (~ 1/month) , we should get it sooner rather than
> later. And following our tooling - we will regenerate and rebuild our
> images once this is available (we are planning to automate this part soon).
> And I hope most or all of those will be addressed.
>
> Following your analogy, that's indeed very true that the images age like
> milk, so it looks like you are supposed to replace it with a fresh bottle
> from time to time. I will try to capture that as best practice.
>
> I am tempted to put your analogy to the proposal ;) I wonder whether the
> Board shares the same sense of humour.
>
> J.
>
>
>
>
> On Tue, Sep 8, 2020 at 12:36 PM Mark J Cox  wrote:
>
> > On Mon, Sep 7, 2020 at 2:21 PM Jarek Potiuk  wrote:
> >
> > > I also talked to the Apache Security team today (there was an issue
> > raised
> > > about the security of the images which I think should be part of the
> > policy
> > > as well.
> > >
> >
> > Thanks Jarek.  What happened is that we got a report to
> > secur...@apache.org
> > about a docker container that when scanned showed a lot of "unfixed
> > vulnerabilities". I'm using quotes there because our usual response to
> > people sending us unfiltered reports from scanning tools is to reject them;
> > we get them quite often outside of containers and binary distributions, and
> > they very rarely are useful.  It's also fairly likely that the majority of
> > the reported issues in the container are completely irrelevant too.  For
> > example the list contained a CVE for a power9 gcc issue.  These scanners
> > are basically going to just report on all the things in the underlying base
> > image that are not updated, and even if you recreated images every day
> > you'd still have unfixed CVEs on the list.
> >
> > Containers and other similar non-source distributions don't age well (a
> > colleague used to say they 'age like milk, not wine'), they'll collect more
> > and more of these layer vulnerabilities over time, and although most will
> > be irrelevant, there are going to be times when such a vulnerability does
> > actually matter, and we need to make sure projects producing them have a
> > process for tracking that either my monitoring (lots of effort) or by at
> > least frequent rebasing to keep them fresh.
> >
> > That's all assuming projects are making good security decisions to start
> > with; basing on images that are maintained, in life, and updated, making
> > sure users know the state/freshness of them, making sure users realise
> > there will be vulns in the underlying layers and how to escalate reporting
> > vulns they find that actually are exposed to the project.  That should all
> > be part of some guidelines on images.
> >
> > Cheers, Mark
> > ASF Security Team
> >
>
>
> --
> +48 660 796 129



-- 
Matt Sicker 

-
To unsubscribe, e-mail: dev-unsubscr...@community.apache.org
For additional commands, e-mail: dev-h...@community.apache.org

Re: Migration and consolidation helm charts for ASF projects from helm/charts to apache/charts git

[Jarek Potiuk]

Will definitely include that in my proposal Mark!

BTW. Speaking of the report you got, we got the user talking to us on
slack, and got the user to retest it on the refreshed image.

It all boiled down to 4 "undefined" risk issues reported by the tool (seems
that their - reasonable - approach is that anything High or Undefined is a
blocker). And the root cause in this case is that the base image that we
used (python-debian-buster) contains those vulnerabilities. Most have been
fixed in other releases of Debian (stretch, bullseye), but the current
(buster) LTS patch has been released over a month ago (3rd of August). With
their release cadence (~ 1/month) , we should get it sooner rather than
later. And following our tooling - we will regenerate and rebuild our
images once this is available (we are planning to automate this part soon).
And I hope most or all of those will be addressed.

Following your analogy, that's indeed very true that the images age like
milk, so it looks like you are supposed to replace it with a fresh bottle
from time to time. I will try to capture that as best practice.

I am tempted to put your analogy to the proposal ;) I wonder whether the
Board shares the same sense of humour.

J.




On Tue, Sep 8, 2020 at 12:36 PM Mark J Cox  wrote:

> On Mon, Sep 7, 2020 at 2:21 PM Jarek Potiuk  wrote:
>
> > I also talked to the Apache Security team today (there was an issue
> raised
> > about the security of the images which I think should be part of the
> policy
> > as well.
> >
>
> Thanks Jarek.  What happened is that we got a report to
> secur...@apache.org
> about a docker container that when scanned showed a lot of "unfixed
> vulnerabilities". I'm using quotes there because our usual response to
> people sending us unfiltered reports from scanning tools is to reject them;
> we get them quite often outside of containers and binary distributions, and
> they very rarely are useful.  It's also fairly likely that the majority of
> the reported issues in the container are completely irrelevant too.  For
> example the list contained a CVE for a power9 gcc issue.  These scanners
> are basically going to just report on all the things in the underlying base
> image that are not updated, and even if you recreated images every day
> you'd still have unfixed CVEs on the list.
>
> Containers and other similar non-source distributions don't age well (a
> colleague used to say they 'age like milk, not wine'), they'll collect more
> and more of these layer vulnerabilities over time, and although most will
> be irrelevant, there are going to be times when such a vulnerability does
> actually matter, and we need to make sure projects producing them have a
> process for tracking that either my monitoring (lots of effort) or by at
> least frequent rebasing to keep them fresh.
>
> That's all assuming projects are making good security decisions to start
> with; basing on images that are maintained, in life, and updated, making
> sure users know the state/freshness of them, making sure users realise
> there will be vulns in the underlying layers and how to escalate reporting
> vulns they find that actually are exposed to the project.  That should all
> be part of some guidelines on images.
>
> Cheers, Mark
> ASF Security Team
>


-- 
+48 660 796 129

Re: Migration and consolidation helm charts for ASF projects from helm/charts to apache/charts git

[Mark J Cox]

On Mon, Sep 7, 2020 at 2:21 PM Jarek Potiuk  wrote:

> I also talked to the Apache Security team today (there was an issue raised
> about the security of the images which I think should be part of the policy
> as well.
>

Thanks Jarek.  What happened is that we got a report to secur...@apache.org
about a docker container that when scanned showed a lot of "unfixed
vulnerabilities". I'm using quotes there because our usual response to
people sending us unfiltered reports from scanning tools is to reject them;
we get them quite often outside of containers and binary distributions, and
they very rarely are useful.  It's also fairly likely that the majority of
the reported issues in the container are completely irrelevant too.  For
example the list contained a CVE for a power9 gcc issue.  These scanners
are basically going to just report on all the things in the underlying base
image that are not updated, and even if you recreated images every day
you'd still have unfixed CVEs on the list.

Containers and other similar non-source distributions don't age well (a
colleague used to say they 'age like milk, not wine'), they'll collect more
and more of these layer vulnerabilities over time, and although most will
be irrelevant, there are going to be times when such a vulnerability does
actually matter, and we need to make sure projects producing them have a
process for tracking that either my monitoring (lots of effort) or by at
least frequent rebasing to keep them fresh.

That's all assuming projects are making good security decisions to start
with; basing on images that are maintained, in life, and updated, making
sure users know the state/freshness of them, making sure users realise
there will be vulns in the underlying layers and how to escalate reporting
vulns they find that actually are exposed to the project.  That should all
be part of some guidelines on images.

Cheers, Mark
ASF Security Team

Re: Migration and consolidation helm charts for ASF projects from helm/charts to apache/charts git

[Jarek Potiuk]

Fantastic, and very actionable! Thanks Bertrand!

I am happy to make a draft of a proposal (I will make sure to mark the
parts that I know are controversial as such) and start the discussion
around this.

I also talked to the Apache Security team today (there was an issue raised
about the security of the images which I think should be part of the policy
as well.

Glad to help with moving it forward.

J.


On Mon, Sep 7, 2020 at 10:24 AM Bertrand Delacretaz 
wrote:

> On Sun, Sep 6, 2020 at 10:37 AM Jarek Potiuk  wrote:
> ...
> > How can I help to make it happen?..
>
> It looks like what you need might require some clarifications of ASF
> policies and probably infrastructure support as well.
>
> IMO, the best way to make these things happen is to create concrete
> proposals that the ASF (Board, infrastructure, Members if needed) can
> approve - modifications or clarifications to existing policy, requests
> for infrastructure services etc.
>
> https://cwiki.apache.org/confluence/display/COMDEV/ComDev+Wiki is a
> good place to prepare those, but any open collaboration place works. A
> https://issues.apache.org/jira/projects/COMDEV ticket might also help
> keep track of progress.
>
> I think the following input can be useful:
>
> * http://apache.org/legal/release-policy.html
> * https://infra.apache.org/how-to-mirror.html
> * https://issues.apache.org/jira/browse/ASFP-23 (accessible to ASF
> Members) has links to related discussions on distributing Compiled
> Packages - nothing secret but some of those links are to internal ASF
> mailing lists.
>
> -Bertrand
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@community.apache.org
> For additional commands, e-mail: dev-h...@community.apache.org
>
>

-- 
+48 660 796 129

Re: Migration and consolidation helm charts for ASF projects from helm/charts to apache/charts git

[Bertrand Delacretaz]

On Sun, Sep 6, 2020 at 10:37 AM Jarek Potiuk  wrote:
...
> How can I help to make it happen?..

It looks like what you need might require some clarifications of ASF
policies and probably infrastructure support as well.

IMO, the best way to make these things happen is to create concrete
proposals that the ASF (Board, infrastructure, Members if needed) can
approve - modifications or clarifications to existing policy, requests
for infrastructure services etc.

https://cwiki.apache.org/confluence/display/COMDEV/ComDev+Wiki is a
good place to prepare those, but any open collaboration place works. A
https://issues.apache.org/jira/projects/COMDEV ticket might also help
keep track of progress.

I think the following input can be useful:

* http://apache.org/legal/release-policy.html
* https://infra.apache.org/how-to-mirror.html
* https://issues.apache.org/jira/browse/ASFP-23 (accessible to ASF
Members) has links to related discussions on distributing Compiled
Packages - nothing secret but some of those links are to internal ASF
mailing lists.

-Bertrand

-
To unsubscribe, e-mail: dev-unsubscr...@community.apache.org
For additional commands, e-mail: dev-h...@community.apache.org

Re: Migration and consolidation helm charts for ASF projects from helm/charts to apache/charts git

[Kaxil Naik]

"The bits we build on top of external components becomes the Apache project
that is ALv2 and is distributed through source archives with convenience
binaries (in this
case, the image layers and YAML files)."

I like this statement from Matt and completely agree with it.

On Sun, Sep 6, 2020, 09:40 Kaxil Naik  wrote:

> I will be happy to help too, definitely and important topic where we need
> a clear guidance on it (what sources and binaries are allowed etc)
>
> Regards,
> Kaxil
> Airflow PMC
>
>
>
> On Sun, Sep 6, 2020, 09:37 Jarek Potiuk  wrote:
>
>> Hey Dave,
>>
>> Thanks for this. Having the ComDev/Infra repository where PMC can publish
>> approved "Chart" and clear docs on what it means to follow the Release
>> Policy is exactly what we need I think.
>>
>> We have a heated debate within Airflow PMC and despite exchanging many
>> messages and (sometimes heated) arguments we remain deeply divided on
>> that.
>> For me, that's a clear sign that we need the ASF help to resolve it.
>>
>> How can I help to make it happen?
>>
>> I am happy to devote quite a lot of my time - both infrastructure and
>> policy discussions/writing as I think this is a very important subject to
>> at least some commercial users of ASF software (and ASF being
>> 'commercially
>> friendly').
>>
>> J.
>>
>>
>>
>> On Sun, Sep 6, 2020 at 1:19 AM Kaxil Naik  wrote:
>>
>> > Great, thanks Dave
>> >
>> > On Sun, Sep 6, 2020, 00:17 Dave Fisher  wrote:
>> >
>> > > Every release must be made in SVN on dist.apache.org. All other
>> channels
>> > > are optional and may or may not have Infra support.
>> > >
>> > > I view this as a request to have ComDev or Infra manage a
>> Gitbox/Github
>> > > repository where any PMC can publish a PMC approved Helm Chart release
>> > from
>> > > their project.
>> > >
>> > > How to make a Helm Chart compliant with Apache Release Policy would
>> part
>> > > of the documentation.
>> > >
>> > > Sent from my iPhone
>> > >
>> > > > On Sep 5, 2020, at 3:30 PM, Kaxil Naik  wrote:
>> > > >
>> > > > 
>> > > >>
>> > > >> The point is that we MUST give the users possibility to (re)build
>> > those
>> > > > images on their own if they want. We
>> > > >
>> > > >
>> > > > From
>> http://www.apache.org/legal/release-policy.html#source-packages,
>> > it
>> > > > does not say it has to be on PyPI. I will reiterate what I said in
>> the
>> > > > Github issue, having a binary on PyPI is no guarantee that it will
>> stay
>> > > > there. Having a binary on Github releases
>> > > > https://github.com/jbub/pgbouncer_exporter/tags also does not
>> > guarantee
>> > > it
>> > > > would not change. If the license are correct and the source is
>> > available
>> > > I
>> > > > definitely do not see this as a problem:
>> > > >
>> > > > Every ASF release MUST contain one or more source packages, which
>> MUST
>> > be
>> > > >> sufficient for a user to build and test the release provided they
>> have
>> > > >> access to the appropriate platform and tools.
>> > > >
>> > > >
>> > > >
>> > > >> On Sat, Sep 5, 2020 at 10:33 PM Jarek Potiuk 
>> > wrote:
>> > > >>
>> > > >> @Matt Sicker  -> agree on both, central chart
>> repo,
>> > > and
>> > > >> per-project sources for charts. Though packaging is a bit different
>> > than
>> > > >> Docker though. The packages contain tar.gz sources of the chart
>> (which
>> > > >> usually includes quite complex go templating logic, not just simple
>> > yaml
>> > > >> files)  and those templates include names of the images used. This
>> > makes
>> > > >> the Helm chart much more close to the -src.tar.gz files we release
>> now
>> > > than
>> > > >> to Docker Images we publish. So IMHO the .tgz part should follow
>> the
>> > > normal
>> > > >> release process with voting @Kaxilnaik .
>> > > >> Unless those are the very same sources of product that are already
>> > > formally
>> > > >> released. But then @Gaetan Semet previously pointed out as bad
>> > practice
>> > > >> to bind the two together.
>> > > >> IMHO that means that we have to vote on each Chart release. But I'd
>> > love
>> > > >> what others think - knowing this packaging details.
>> > > >>
>> > > >> @ermanndimb- my point for images pointed to by the helm chart is
>> not
>> > > about
>> > > >> copying full sources (I think from this discussion I realised where
>> > the
>> > > >> main confusion was) . This is not needed. I just made a PR to show
>> how
>> > > this
>> > > >> can be done with the pgbouncer-exporter.  This is quite a good
>> example
>> > > to
>> > > >> base our discussion on:
>> https://github.com/apache/airflow/pull/10759
>> > > >>
>> > > >> The point is that we MUST give the users possibility to (re)build
>> > those
>> > > >> images on their own if they want. They should be able to do that
>> > using:
>> > > >>
>> > > >>  a) official base images (DockerHub has the "official image
>> program"
>> > for
>> > > >> platforms. openjdk, python,debian, alpine - those are all official
>> > > images
>> > > >> in DockerHub terms
>> > > >>  b) ha

Re: Migration and consolidation helm charts for ASF projects from helm/charts to apache/charts git

[Kaxil Naik]

I will be happy to help too, definitely and important topic where we need a
clear guidance on it (what sources and binaries are allowed etc)

Regards,
Kaxil
Airflow PMC



On Sun, Sep 6, 2020, 09:37 Jarek Potiuk  wrote:

> Hey Dave,
>
> Thanks for this. Having the ComDev/Infra repository where PMC can publish
> approved "Chart" and clear docs on what it means to follow the Release
> Policy is exactly what we need I think.
>
> We have a heated debate within Airflow PMC and despite exchanging many
> messages and (sometimes heated) arguments we remain deeply divided on that.
> For me, that's a clear sign that we need the ASF help to resolve it.
>
> How can I help to make it happen?
>
> I am happy to devote quite a lot of my time - both infrastructure and
> policy discussions/writing as I think this is a very important subject to
> at least some commercial users of ASF software (and ASF being 'commercially
> friendly').
>
> J.
>
>
>
> On Sun, Sep 6, 2020 at 1:19 AM Kaxil Naik  wrote:
>
> > Great, thanks Dave
> >
> > On Sun, Sep 6, 2020, 00:17 Dave Fisher  wrote:
> >
> > > Every release must be made in SVN on dist.apache.org. All other
> channels
> > > are optional and may or may not have Infra support.
> > >
> > > I view this as a request to have ComDev or Infra manage a Gitbox/Github
> > > repository where any PMC can publish a PMC approved Helm Chart release
> > from
> > > their project.
> > >
> > > How to make a Helm Chart compliant with Apache Release Policy would
> part
> > > of the documentation.
> > >
> > > Sent from my iPhone
> > >
> > > > On Sep 5, 2020, at 3:30 PM, Kaxil Naik  wrote:
> > > >
> > > > 
> > > >>
> > > >> The point is that we MUST give the users possibility to (re)build
> > those
> > > > images on their own if they want. We
> > > >
> > > >
> > > > From http://www.apache.org/legal/release-policy.html#source-packages
> ,
> > it
> > > > does not say it has to be on PyPI. I will reiterate what I said in
> the
> > > > Github issue, having a binary on PyPI is no guarantee that it will
> stay
> > > > there. Having a binary on Github releases
> > > > https://github.com/jbub/pgbouncer_exporter/tags also does not
> > guarantee
> > > it
> > > > would not change. If the license are correct and the source is
> > available
> > > I
> > > > definitely do not see this as a problem:
> > > >
> > > > Every ASF release MUST contain one or more source packages, which
> MUST
> > be
> > > >> sufficient for a user to build and test the release provided they
> have
> > > >> access to the appropriate platform and tools.
> > > >
> > > >
> > > >
> > > >> On Sat, Sep 5, 2020 at 10:33 PM Jarek Potiuk 
> > wrote:
> > > >>
> > > >> @Matt Sicker  -> agree on both, central chart
> repo,
> > > and
> > > >> per-project sources for charts. Though packaging is a bit different
> > than
> > > >> Docker though. The packages contain tar.gz sources of the chart
> (which
> > > >> usually includes quite complex go templating logic, not just simple
> > yaml
> > > >> files)  and those templates include names of the images used. This
> > makes
> > > >> the Helm chart much more close to the -src.tar.gz files we release
> now
> > > than
> > > >> to Docker Images we publish. So IMHO the .tgz part should follow the
> > > normal
> > > >> release process with voting @Kaxilnaik .
> > > >> Unless those are the very same sources of product that are already
> > > formally
> > > >> released. But then @Gaetan Semet previously pointed out as bad
> > practice
> > > >> to bind the two together.
> > > >> IMHO that means that we have to vote on each Chart release. But I'd
> > love
> > > >> what others think - knowing this packaging details.
> > > >>
> > > >> @ermanndimb- my point for images pointed to by the helm chart is not
> > > about
> > > >> copying full sources (I think from this discussion I realised where
> > the
> > > >> main confusion was) . This is not needed. I just made a PR to show
> how
> > > this
> > > >> can be done with the pgbouncer-exporter.  This is quite a good
> example
> > > to
> > > >> base our discussion on:
> https://github.com/apache/airflow/pull/10759
> > > >>
> > > >> The point is that we MUST give the users possibility to (re)build
> > those
> > > >> images on their own if they want. They should be able to do that
> > using:
> > > >>
> > > >>  a) official base images (DockerHub has the "official image program"
> > for
> > > >> platforms. openjdk, python,debian, alpine - those are all official
> > > images
> > > >> in DockerHub terms
> > > >>  b) having access to properly licensed source code with some
> > guarantees
> > > >> that it will not disappear ("official" repo, multiple forks in
> Github,
> > > >> heavily used by others, fork under "apache" organisation) .
> > > >>  c) instructions where to find the sources and build the images
> > > >> (Dockerfile + necessary scripts to build the image if in case it is
> > not
> > > >> straightforward). Similarly as we provide instructions on building
> the
> > > >> "product".
>

Re: Migration and consolidation helm charts for ASF projects from helm/charts to apache/charts git

[Jarek Potiuk]

Hey Dave,

Thanks for this. Having the ComDev/Infra repository where PMC can publish
approved "Chart" and clear docs on what it means to follow the Release
Policy is exactly what we need I think.

We have a heated debate within Airflow PMC and despite exchanging many
messages and (sometimes heated) arguments we remain deeply divided on that.
For me, that's a clear sign that we need the ASF help to resolve it.

How can I help to make it happen?

I am happy to devote quite a lot of my time - both infrastructure and
policy discussions/writing as I think this is a very important subject to
at least some commercial users of ASF software (and ASF being 'commercially
friendly').

J.



On Sun, Sep 6, 2020 at 1:19 AM Kaxil Naik  wrote:

> Great, thanks Dave
>
> On Sun, Sep 6, 2020, 00:17 Dave Fisher  wrote:
>
> > Every release must be made in SVN on dist.apache.org. All other channels
> > are optional and may or may not have Infra support.
> >
> > I view this as a request to have ComDev or Infra manage a Gitbox/Github
> > repository where any PMC can publish a PMC approved Helm Chart release
> from
> > their project.
> >
> > How to make a Helm Chart compliant with Apache Release Policy would part
> > of the documentation.
> >
> > Sent from my iPhone
> >
> > > On Sep 5, 2020, at 3:30 PM, Kaxil Naik  wrote:
> > >
> > > 
> > >>
> > >> The point is that we MUST give the users possibility to (re)build
> those
> > > images on their own if they want. We
> > >
> > >
> > > From http://www.apache.org/legal/release-policy.html#source-packages,
> it
> > > does not say it has to be on PyPI. I will reiterate what I said in the
> > > Github issue, having a binary on PyPI is no guarantee that it will stay
> > > there. Having a binary on Github releases
> > > https://github.com/jbub/pgbouncer_exporter/tags also does not
> guarantee
> > it
> > > would not change. If the license are correct and the source is
> available
> > I
> > > definitely do not see this as a problem:
> > >
> > > Every ASF release MUST contain one or more source packages, which MUST
> be
> > >> sufficient for a user to build and test the release provided they have
> > >> access to the appropriate platform and tools.
> > >
> > >
> > >
> > >> On Sat, Sep 5, 2020 at 10:33 PM Jarek Potiuk 
> wrote:
> > >>
> > >> @Matt Sicker  -> agree on both, central chart repo,
> > and
> > >> per-project sources for charts. Though packaging is a bit different
> than
> > >> Docker though. The packages contain tar.gz sources of the chart (which
> > >> usually includes quite complex go templating logic, not just simple
> yaml
> > >> files)  and those templates include names of the images used. This
> makes
> > >> the Helm chart much more close to the -src.tar.gz files we release now
> > than
> > >> to Docker Images we publish. So IMHO the .tgz part should follow the
> > normal
> > >> release process with voting @Kaxilnaik .
> > >> Unless those are the very same sources of product that are already
> > formally
> > >> released. But then @Gaetan Semet previously pointed out as bad
> practice
> > >> to bind the two together.
> > >> IMHO that means that we have to vote on each Chart release. But I'd
> love
> > >> what others think - knowing this packaging details.
> > >>
> > >> @ermanndimb- my point for images pointed to by the helm chart is not
> > about
> > >> copying full sources (I think from this discussion I realised where
> the
> > >> main confusion was) . This is not needed. I just made a PR to show how
> > this
> > >> can be done with the pgbouncer-exporter.  This is quite a good example
> > to
> > >> base our discussion on: https://github.com/apache/airflow/pull/10759
> > >>
> > >> The point is that we MUST give the users possibility to (re)build
> those
> > >> images on their own if they want. They should be able to do that
> using:
> > >>
> > >>  a) official base images (DockerHub has the "official image program"
> for
> > >> platforms. openjdk, python,debian, alpine - those are all official
> > images
> > >> in DockerHub terms
> > >>  b) having access to properly licensed source code with some
> guarantees
> > >> that it will not disappear ("official" repo, multiple forks in Github,
> > >> heavily used by others, fork under "apache" organisation) .
> > >>  c) instructions where to find the sources and build the images
> > >> (Dockerfile + necessary scripts to build the image if in case it is
> not
> > >> straightforward). Similarly as we provide instructions on building the
> > >> "product".
> > >>
> > >> If the first two are available but it's not obvious how the image was
> > built
> > >> - IMHO we need to provide the user instructions on how to build those
> > >> images (ideally a script that does it automatically).
> > >>
> > >> This is all fulfilled by the PR above as an example:
> > >> a) based on official, alpine image
> > >> b) the pgbouncer code is properly licenced and we forked the sources
> > >> several times to be sure it won't disappear
> > >> c) we have a scri

Re: Migration and consolidation helm charts for ASF projects from helm/charts to apache/charts git

[Kaxil Naik]

Great, thanks Dave

On Sun, Sep 6, 2020, 00:17 Dave Fisher  wrote:

> Every release must be made in SVN on dist.apache.org. All other channels
> are optional and may or may not have Infra support.
>
> I view this as a request to have ComDev or Infra manage a Gitbox/Github
> repository where any PMC can publish a PMC approved Helm Chart release from
> their project.
>
> How to make a Helm Chart compliant with Apache Release Policy would part
> of the documentation.
>
> Sent from my iPhone
>
> > On Sep 5, 2020, at 3:30 PM, Kaxil Naik  wrote:
> >
> > 
> >>
> >> The point is that we MUST give the users possibility to (re)build those
> > images on their own if they want. We
> >
> >
> > From http://www.apache.org/legal/release-policy.html#source-packages, it
> > does not say it has to be on PyPI. I will reiterate what I said in the
> > Github issue, having a binary on PyPI is no guarantee that it will stay
> > there. Having a binary on Github releases
> > https://github.com/jbub/pgbouncer_exporter/tags also does not guarantee
> it
> > would not change. If the license are correct and the source is available
> I
> > definitely do not see this as a problem:
> >
> > Every ASF release MUST contain one or more source packages, which MUST be
> >> sufficient for a user to build and test the release provided they have
> >> access to the appropriate platform and tools.
> >
> >
> >
> >> On Sat, Sep 5, 2020 at 10:33 PM Jarek Potiuk  wrote:
> >>
> >> @Matt Sicker  -> agree on both, central chart repo,
> and
> >> per-project sources for charts. Though packaging is a bit different than
> >> Docker though. The packages contain tar.gz sources of the chart (which
> >> usually includes quite complex go templating logic, not just simple yaml
> >> files)  and those templates include names of the images used. This makes
> >> the Helm chart much more close to the -src.tar.gz files we release now
> than
> >> to Docker Images we publish. So IMHO the .tgz part should follow the
> normal
> >> release process with voting @Kaxilnaik .
> >> Unless those are the very same sources of product that are already
> formally
> >> released. But then @Gaetan Semet previously pointed out as bad practice
> >> to bind the two together.
> >> IMHO that means that we have to vote on each Chart release. But I'd love
> >> what others think - knowing this packaging details.
> >>
> >> @ermanndimb- my point for images pointed to by the helm chart is not
> about
> >> copying full sources (I think from this discussion I realised where the
> >> main confusion was) . This is not needed. I just made a PR to show how
> this
> >> can be done with the pgbouncer-exporter.  This is quite a good example
> to
> >> base our discussion on: https://github.com/apache/airflow/pull/10759
> >>
> >> The point is that we MUST give the users possibility to (re)build those
> >> images on their own if they want. They should be able to do that using:
> >>
> >>  a) official base images (DockerHub has the "official image program" for
> >> platforms. openjdk, python,debian, alpine - those are all official
> images
> >> in DockerHub terms
> >>  b) having access to properly licensed source code with some guarantees
> >> that it will not disappear ("official" repo, multiple forks in Github,
> >> heavily used by others, fork under "apache" organisation) .
> >>  c) instructions where to find the sources and build the images
> >> (Dockerfile + necessary scripts to build the image if in case it is not
> >> straightforward). Similarly as we provide instructions on building the
> >> "product".
> >>
> >> If the first two are available but it's not obvious how the image was
> built
> >> - IMHO we need to provide the user instructions on how to build those
> >> images (ideally a script that does it automatically).
> >>
> >> This is all fulfilled by the PR above as an example:
> >> a) based on official, alpine image
> >> b) the pgbouncer code is properly licenced and we forked the sources
> >> several times to be sure it won't disappear
> >> c) we have a script that builds and publishes the image (image is
> published
> >> under apache/airflow DockerHub)
> >>
> >> IMHO - that's a very good example to show the kind of approach we can
> take,
> >> and shows that it isn't difficult to handle it well at all.
> >>
> >> The alternative that I am afraid of is - that we require our users to
> use
> >> binaries which are not "official" but maintained and controlled by
> >> 3rd-party (without a straightforward way of building the binary from
> >> official images + properly licensed sources + instructions). If we do
> not
> >> give those to the user but we simply point to a 3rd-party binary bimage
> -
> >> we both endorse the 3rd party, and introduce dependency on the
> 3rd-party.
> >> The user then has to rely on the 3rd-party to use the Airflow Helm
> Chart -
> >> which I think is a very bad idea.
> >>
> >> J.
> >>
> >>
> >>> On Sat, Sep 5, 2020 at 7:42 PM Kaxil Naik  wrote:
> >>>
> >>> Does voting needs to ha

Re: Migration and consolidation helm charts for ASF projects from helm/charts to apache/charts git

[Dave Fisher]

Every release must be made in SVN on dist.apache.org. All other channels are 
optional and may or may not have Infra support.

I view this as a request to have ComDev or Infra manage a Gitbox/Github 
repository where any PMC can publish a PMC approved Helm Chart release from 
their project.

How to make a Helm Chart compliant with Apache Release Policy would part of the 
documentation.

Sent from my iPhone

> On Sep 5, 2020, at 3:30 PM, Kaxil Naik  wrote:
> 
> 
>> 
>> The point is that we MUST give the users possibility to (re)build those
> images on their own if they want. We
> 
> 
> From http://www.apache.org/legal/release-policy.html#source-packages, it
> does not say it has to be on PyPI. I will reiterate what I said in the
> Github issue, having a binary on PyPI is no guarantee that it will stay
> there. Having a binary on Github releases
> https://github.com/jbub/pgbouncer_exporter/tags also does not guarantee it
> would not change. If the license are correct and the source is available I
> definitely do not see this as a problem:
> 
> Every ASF release MUST contain one or more source packages, which MUST be
>> sufficient for a user to build and test the release provided they have
>> access to the appropriate platform and tools.
> 
> 
> 
>> On Sat, Sep 5, 2020 at 10:33 PM Jarek Potiuk  wrote:
>> 
>> @Matt Sicker  -> agree on both, central chart repo, and
>> per-project sources for charts. Though packaging is a bit different than
>> Docker though. The packages contain tar.gz sources of the chart (which
>> usually includes quite complex go templating logic, not just simple yaml
>> files)  and those templates include names of the images used. This makes
>> the Helm chart much more close to the -src.tar.gz files we release now than
>> to Docker Images we publish. So IMHO the .tgz part should follow the normal
>> release process with voting @Kaxilnaik .
>> Unless those are the very same sources of product that are already formally
>> released. But then @Gaetan Semet previously pointed out as bad practice
>> to bind the two together.
>> IMHO that means that we have to vote on each Chart release. But I'd love
>> what others think - knowing this packaging details.
>> 
>> @ermanndimb- my point for images pointed to by the helm chart is not about
>> copying full sources (I think from this discussion I realised where the
>> main confusion was) . This is not needed. I just made a PR to show how this
>> can be done with the pgbouncer-exporter.  This is quite a good example to
>> base our discussion on: https://github.com/apache/airflow/pull/10759
>> 
>> The point is that we MUST give the users possibility to (re)build those
>> images on their own if they want. They should be able to do that using:
>> 
>>  a) official base images (DockerHub has the "official image program" for
>> platforms. openjdk, python,debian, alpine - those are all official images
>> in DockerHub terms
>>  b) having access to properly licensed source code with some guarantees
>> that it will not disappear ("official" repo, multiple forks in Github,
>> heavily used by others, fork under "apache" organisation) .
>>  c) instructions where to find the sources and build the images
>> (Dockerfile + necessary scripts to build the image if in case it is not
>> straightforward). Similarly as we provide instructions on building the
>> "product".
>> 
>> If the first two are available but it's not obvious how the image was built
>> - IMHO we need to provide the user instructions on how to build those
>> images (ideally a script that does it automatically).
>> 
>> This is all fulfilled by the PR above as an example:
>> a) based on official, alpine image
>> b) the pgbouncer code is properly licenced and we forked the sources
>> several times to be sure it won't disappear
>> c) we have a script that builds and publishes the image (image is published
>> under apache/airflow DockerHub)
>> 
>> IMHO - that's a very good example to show the kind of approach we can take,
>> and shows that it isn't difficult to handle it well at all.
>> 
>> The alternative that I am afraid of is - that we require our users to use
>> binaries which are not "official" but maintained and controlled by
>> 3rd-party (without a straightforward way of building the binary from
>> official images + properly licensed sources + instructions). If we do not
>> give those to the user but we simply point to a 3rd-party binary bimage -
>> we both endorse the 3rd party, and introduce dependency on the 3rd-party.
>> The user then has to rely on the 3rd-party to use the Airflow Helm Chart -
>> which I think is a very bad idea.
>> 
>> J.
>> 
>> 
>>> On Sat, Sep 5, 2020 at 7:42 PM Kaxil Naik  wrote:
>>> 
>>> Does voting needs to happen for "releasing" the Helm chart ? Do we any
>>> guidelines from the ASF around releasing Helm Chart & Dockerfile?
>>> 
 On Sat, Sep 5, 2020, 17:36 Matt Sicker  wrote:
>>> 
 If packaging is similar to Docker, then our Helm charts will still
 need some third p

Re: Migration and consolidation helm charts for ASF projects from helm/charts to apache/charts git

[Kaxil Naik]

>The point is that we MUST give the users possibility to (re)build those
images on their own if they want.


>From http://www.apache.org/legal/release-policy.html#source-packages, it
does not say it has to be on PyPI. I will reiterate what I said in the
Github issue, having a binary on PyPI is no guarantee that it will stay
there. Having a binary on Github releases
https://github.com/jbub/pgbouncer_exporter/tags also does not guarantee it
would not change. If the license are correct and the source is available I
definitely do not see this as a problem:

Every ASF release MUST contain one or more source packages, which MUST be
> sufficient for a user to build and test the release provided they have
> access to the appropriate platform and tools.



On Sat, Sep 5, 2020 at 10:33 PM Jarek Potiuk  wrote:

> @Matt Sicker  -> agree on both, central chart repo, and
> per-project sources for charts. Though packaging is a bit different than
> Docker though. The packages contain tar.gz sources of the chart (which
> usually includes quite complex go templating logic, not just simple yaml
> files)  and those templates include names of the images used. This makes
> the Helm chart much more close to the -src.tar.gz files we release now than
> to Docker Images we publish. So IMHO the .tgz part should follow the normal
> release process with voting @Kaxilnaik .
> Unless those are the very same sources of product that are already formally
> released. But then @Gaetan Semet previously pointed out as bad practice
> to bind the two together.
>  IMHO that means that we have to vote on each Chart release. But I'd love
> what others think - knowing this packaging details.
>
> @ermanndimb- my point for images pointed to by the helm chart is not about
> copying full sources (I think from this discussion I realised where the
> main confusion was) . This is not needed. I just made a PR to show how this
> can be done with the pgbouncer-exporter.  This is quite a good example to
> base our discussion on: https://github.com/apache/airflow/pull/10759
>
> The point is that we MUST give the users possibility to (re)build those
> images on their own if they want. They should be able to do that using:
>
>   a) official base images (DockerHub has the "official image program" for
> platforms. openjdk, python,debian, alpine - those are all official images
> in DockerHub terms
>   b) having access to properly licensed source code with some guarantees
> that it will not disappear ("official" repo, multiple forks in Github,
> heavily used by others, fork under "apache" organisation) .
>   c) instructions where to find the sources and build the images
> (Dockerfile + necessary scripts to build the image if in case it is not
> straightforward). Similarly as we provide instructions on building the
> "product".
>
> If the first two are available but it's not obvious how the image was built
> - IMHO we need to provide the user instructions on how to build those
> images (ideally a script that does it automatically).
>
> This is all fulfilled by the PR above as an example:
> a) based on official, alpine image
> b) the pgbouncer code is properly licenced and we forked the sources
> several times to be sure it won't disappear
> c) we have a script that builds and publishes the image (image is published
> under apache/airflow DockerHub)
>
> IMHO - that's a very good example to show the kind of approach we can take,
> and shows that it isn't difficult to handle it well at all.
>
> The alternative that I am afraid of is - that we require our users to use
> binaries which are not "official" but maintained and controlled by
> 3rd-party (without a straightforward way of building the binary from
> official images + properly licensed sources + instructions). If we do not
> give those to the user but we simply point to a 3rd-party binary bimage -
> we both endorse the 3rd party, and introduce dependency on the 3rd-party.
> The user then has to rely on the 3rd-party to use the Airflow Helm Chart -
> which I think is a very bad idea.
>
> J.
>
>
> On Sat, Sep 5, 2020 at 7:42 PM Kaxil Naik  wrote:
>
> > Does voting needs to happen for "releasing" the Helm chart ? Do we any
> > guidelines from the ASF around releasing Helm Chart & Dockerfile?
> >
> > On Sat, Sep 5, 2020, 17:36 Matt Sicker  wrote:
> >
> > > If packaging is similar to Docker, then our Helm charts will still
> > > need some third party base images and such for GPL licensed system
> > > dependencies (besides the kernel itself, there's OpenJDK which is
> > > definitely used in plenty of projects here). The bits we build on top
> > > of external components becomes the Apache project that is ALv2 and is
> > > distributed through source archives with convenience binaries (in this
> > > case, the image layers and YAML files).
> > >
> > > I do like the idea of having a central Apache Helm chart repo where
> > > releases can be published. That makes it simpler for users to combine
> > > multiple Apache charts tog

Re: Migration and consolidation helm charts for ASF projects from helm/charts to apache/charts git

[Jarek Potiuk]

@Matt Sicker  -> agree on both, central chart repo, and
per-project sources for charts. Though packaging is a bit different than
Docker though. The packages contain tar.gz sources of the chart (which
usually includes quite complex go templating logic, not just simple yaml
files)  and those templates include names of the images used. This makes
the Helm chart much more close to the -src.tar.gz files we release now than
to Docker Images we publish. So IMHO the .tgz part should follow the normal
release process with voting @Kaxilnaik .
Unless those are the very same sources of product that are already formally
released. But then @Gaetan Semet previously pointed out as bad practice
to bind the two together.
 IMHO that means that we have to vote on each Chart release. But I'd love
what others think - knowing this packaging details.

@ermanndimb- my point for images pointed to by the helm chart is not about
copying full sources (I think from this discussion I realised where the
main confusion was) . This is not needed. I just made a PR to show how this
can be done with the pgbouncer-exporter.  This is quite a good example to
base our discussion on: https://github.com/apache/airflow/pull/10759

The point is that we MUST give the users possibility to (re)build those
images on their own if they want. They should be able to do that using:

  a) official base images (DockerHub has the "official image program" for
platforms. openjdk, python,debian, alpine - those are all official images
in DockerHub terms
  b) having access to properly licensed source code with some guarantees
that it will not disappear ("official" repo, multiple forks in Github,
heavily used by others, fork under "apache" organisation) .
  c) instructions where to find the sources and build the images
(Dockerfile + necessary scripts to build the image if in case it is not
straightforward). Similarly as we provide instructions on building the
"product".

If the first two are available but it's not obvious how the image was built
- IMHO we need to provide the user instructions on how to build those
images (ideally a script that does it automatically).

This is all fulfilled by the PR above as an example:
a) based on official, alpine image
b) the pgbouncer code is properly licenced and we forked the sources
several times to be sure it won't disappear
c) we have a script that builds and publishes the image (image is published
under apache/airflow DockerHub)

IMHO - that's a very good example to show the kind of approach we can take,
and shows that it isn't difficult to handle it well at all.

The alternative that I am afraid of is - that we require our users to use
binaries which are not "official" but maintained and controlled by
3rd-party (without a straightforward way of building the binary from
official images + properly licensed sources + instructions). If we do not
give those to the user but we simply point to a 3rd-party binary bimage -
we both endorse the 3rd party, and introduce dependency on the 3rd-party.
The user then has to rely on the 3rd-party to use the Airflow Helm Chart -
which I think is a very bad idea.

J.


On Sat, Sep 5, 2020 at 7:42 PM Kaxil Naik  wrote:

> Does voting needs to happen for "releasing" the Helm chart ? Do we any
> guidelines from the ASF around releasing Helm Chart & Dockerfile?
>
> On Sat, Sep 5, 2020, 17:36 Matt Sicker  wrote:
>
> > If packaging is similar to Docker, then our Helm charts will still
> > need some third party base images and such for GPL licensed system
> > dependencies (besides the kernel itself, there's OpenJDK which is
> > definitely used in plenty of projects here). The bits we build on top
> > of external components becomes the Apache project that is ALv2 and is
> > distributed through source archives with convenience binaries (in this
> > case, the image layers and YAML files).
> >
> > I do like the idea of having a central Apache Helm chart repo where
> > releases can be published. That makes it simpler for users to combine
> > multiple Apache charts together, too, based on my past experience with
> > using Helm (especially with the new decentralized charts distribution
> > model they've switched to). I do think it makes sense to try and
> > maintain the sources and such for those charts in the individual
> > projects since each PMC may have different workflows on how that's
> > created, and it makes it easier to release alongside the normal
> > project's releases.
> >
> > On Sat, 5 Sep 2020 at 11:13, Daniel Imberman 
> wrote:
> > >
> > > While I understand the necessity for users to have access to all source
> > code, I'm a bit concerned about the complexity this places on the
> > open-source project. Let's take Airflow as an example. If we want to use
> > pgbouncer in our home chart are we expected to be able to build this
> > external project?
> > >
> > > I think that if this is ASF policy, then maybe we need to modify the
> > policy to meet the current ecosystem. Helm charts are made to h

Re: Migration and consolidation helm charts for ASF projects from helm/charts to apache/charts git

[Kaxil Naik]

Does voting needs to happen for "releasing" the Helm chart ? Do we any
guidelines from the ASF around releasing Helm Chart & Dockerfile?

On Sat, Sep 5, 2020, 17:36 Matt Sicker  wrote:

> If packaging is similar to Docker, then our Helm charts will still
> need some third party base images and such for GPL licensed system
> dependencies (besides the kernel itself, there's OpenJDK which is
> definitely used in plenty of projects here). The bits we build on top
> of external components becomes the Apache project that is ALv2 and is
> distributed through source archives with convenience binaries (in this
> case, the image layers and YAML files).
>
> I do like the idea of having a central Apache Helm chart repo where
> releases can be published. That makes it simpler for users to combine
> multiple Apache charts together, too, based on my past experience with
> using Helm (especially with the new decentralized charts distribution
> model they've switched to). I do think it makes sense to try and
> maintain the sources and such for those charts in the individual
> projects since each PMC may have different workflows on how that's
> created, and it makes it easier to release alongside the normal
> project's releases.
>
> On Sat, 5 Sep 2020 at 11:13, Daniel Imberman  wrote:
> >
> > While I understand the necessity for users to have access to all source
> code, I'm a bit concerned about the complexity this places on the
> open-source project. Let's take Airflow as an example. If we want to use
> pgbouncer in our home chart are we expected to be able to build this
> external project?
> >
> > I think that if this is ASF policy, then maybe we need to modify the
> policy to meet the current ecosystem. Helm charts are made to handle
> complex deployments, and expecting a project to own every other piece of
> their ecosystem is a pretty heavy burden. Is there some compromise we can
> come to? Can we store certain pieces such as source code, Dockerfile, and
> docker binary in some Apache cold storage to ensure they are reproduceable?
> >
> > Ideally I'd just like to keep the burden on project maintainers to a
> minimum. This could heavily discourage projects from creating official helm
> charts.
> >
> >
> > On 2020/09/04 15:05:56, Scott Rigby  wrote:
> > > Jarek, this is fantastic! Please let me know if I/we can help with
> this. The Helm team has written tools to make some of these processes
> easier, including steps to preserve former git history, and GitHub Actions
> for CI (chart testing) and CD (releasing chart version packages via GitHub
> release artifacts).
> > >
> > > Bertrand, many orgs are adopting a git repo naming convention
> "helm-charts" to make this explicit.
> > >
> > > Fun follow-up question: a helm repo is really just an index (YAML)
> file of metadata in a format the helm client expects, including links to
> packaged tarballs for each immutable version of a chart. The idea of Apache
> mirror hosting is on the table, one issue we have is that the GCP object
> storage buckets that currently host all previous versions of stable and
> incubator repo charts will be deleted after support for these repos ends on
> Nov 13th (
> https://github.com/helm/charts/blob/master/README.md#deprecation-timeline).
> Would it be possible for Apache to host a mirror of all these packages
> (note, they are all Apache licenced), or only chart packages related to
> Apache software (airflow, solr, spark, etc)?
> > >
> > > Scott
> > >
> > > On 2020/09/04 08:47:30, Jarek Potiuk  wrote:
> > > > Big +1 for doing this within existing ASF infrastructure. Maybe
> indeed
> > > > current SVN  publishing a release snapshot  + mirroring is the best
> way.
> > > >
> > > > IMHO such 'released' charts do not have to have commit history at all
> > > > (except 'release' commits) - just release 'snapshots' is enough. For
> all
> > > > the other development uses, whatever the source repository is, you
> will be
> > > > able too install 'development' chart from the sources.
> > > >
> > > > Still having clear guidelines on how to approach Docker images and
> > > > 'rebuildability' by the users from sources is an important aspect
> IMHO.
> > > > J.
> > > >
> > > > pt., 4 wrz 2020, 10:15 użytkownik Bertrand Delacretaz <
> > > > bdelacre...@apache.org> napisał:
> > > >
> > > > > Hi,
> > > > >
> > > > > On Fri, Sep 4, 2020 at 9:22 AM Jarek Potiuk 
> wrote:
> > > > > > ...I think having ASF "apache/chart" for all "officially
> released" helm
> > > > > > charts is the best solution
> > > > >
> > > > > Sounds like a good idea but please use a more specific name,
> "charts"
> > > > > is very generic.
> > > > >
> > > > > If distributing those Helm charts does not requires more than a web
> > > > > server (with some metadata files I suppose) that can probably go
> in a
> > > > > specific folder under https://dist.apache.org/repos/dist/release/
> > > > >
> > > > > The next question is what kind of traffic is expected there, and
> can
> > > > > the clie

Re: Migration and consolidation helm charts for ASF projects from helm/charts to apache/charts git

[Matt Sicker]

If packaging is similar to Docker, then our Helm charts will still
need some third party base images and such for GPL licensed system
dependencies (besides the kernel itself, there's OpenJDK which is
definitely used in plenty of projects here). The bits we build on top
of external components becomes the Apache project that is ALv2 and is
distributed through source archives with convenience binaries (in this
case, the image layers and YAML files).

I do like the idea of having a central Apache Helm chart repo where
releases can be published. That makes it simpler for users to combine
multiple Apache charts together, too, based on my past experience with
using Helm (especially with the new decentralized charts distribution
model they've switched to). I do think it makes sense to try and
maintain the sources and such for those charts in the individual
projects since each PMC may have different workflows on how that's
created, and it makes it easier to release alongside the normal
project's releases.

On Sat, 5 Sep 2020 at 11:13, Daniel Imberman  wrote:
>
> While I understand the necessity for users to have access to all source code, 
> I'm a bit concerned about the complexity this places on the open-source 
> project. Let's take Airflow as an example. If we want to use pgbouncer in our 
> home chart are we expected to be able to build this external project?
>
> I think that if this is ASF policy, then maybe we need to modify the policy 
> to meet the current ecosystem. Helm charts are made to handle complex 
> deployments, and expecting a project to own every other piece of their 
> ecosystem is a pretty heavy burden. Is there some compromise we can come to? 
> Can we store certain pieces such as source code, Dockerfile, and docker 
> binary in some Apache cold storage to ensure they are reproduceable?
>
> Ideally I'd just like to keep the burden on project maintainers to a minimum. 
> This could heavily discourage projects from creating official helm charts.
>
>
> On 2020/09/04 15:05:56, Scott Rigby  wrote:
> > Jarek, this is fantastic! Please let me know if I/we can help with this. 
> > The Helm team has written tools to make some of these processes easier, 
> > including steps to preserve former git history, and GitHub Actions for CI 
> > (chart testing) and CD (releasing chart version packages via GitHub release 
> > artifacts).
> >
> > Bertrand, many orgs are adopting a git repo naming convention "helm-charts" 
> > to make this explicit.
> >
> > Fun follow-up question: a helm repo is really just an index (YAML) file of 
> > metadata in a format the helm client expects, including links to packaged 
> > tarballs for each immutable version of a chart. The idea of Apache mirror 
> > hosting is on the table, one issue we have is that the GCP object storage 
> > buckets that currently host all previous versions of stable and incubator 
> > repo charts will be deleted after support for these repos ends on Nov 13th 
> > (https://github.com/helm/charts/blob/master/README.md#deprecation-timeline).
> >  Would it be possible for Apache to host a mirror of all these packages 
> > (note, they are all Apache licenced), or only chart packages related to 
> > Apache software (airflow, solr, spark, etc)?
> >
> > Scott
> >
> > On 2020/09/04 08:47:30, Jarek Potiuk  wrote:
> > > Big +1 for doing this within existing ASF infrastructure. Maybe indeed
> > > current SVN  publishing a release snapshot  + mirroring is the best way.
> > >
> > > IMHO such 'released' charts do not have to have commit history at all
> > > (except 'release' commits) - just release 'snapshots' is enough. For all
> > > the other development uses, whatever the source repository is, you will be
> > > able too install 'development' chart from the sources.
> > >
> > > Still having clear guidelines on how to approach Docker images and
> > > 'rebuildability' by the users from sources is an important aspect IMHO.
> > > J.
> > >
> > > pt., 4 wrz 2020, 10:15 użytkownik Bertrand Delacretaz <
> > > bdelacre...@apache.org> napisał:
> > >
> > > > Hi,
> > > >
> > > > On Fri, Sep 4, 2020 at 9:22 AM Jarek Potiuk  wrote:
> > > > > ...I think having ASF "apache/chart" for all "officially released" 
> > > > > helm
> > > > > charts is the best solution
> > > >
> > > > Sounds like a good idea but please use a more specific name, "charts"
> > > > is very generic.
> > > >
> > > > If distributing those Helm charts does not requires more than a web
> > > > server (with some metadata files I suppose) that can probably go in a
> > > > specific folder under https://dist.apache.org/repos/dist/release/
> > > >
> > > > The next question is what kind of traffic is expected there, and can
> > > > the clients which will download those Helm charts handle redirects to
> > > > distribution mirrors? If yes the mechanism of
> > > > http://www.apache.org/dev/release-download-pages.html comes to mind,
> > > > if you can piggyback on that mirroring infrastructure that's a big
> > > > plus IM

Re: Migration and consolidation helm charts for ASF projects from helm/charts to apache/charts git

[Daniel Imberman]

While I understand the necessity for users to have access to all source code, 
I'm a bit concerned about the complexity this places on the open-source 
project. Let's take Airflow as an example. If we want to use pgbouncer in our 
home chart are we expected to be able to build this external project?

I think that if this is ASF policy, then maybe we need to modify the policy to 
meet the current ecosystem. Helm charts are made to handle complex deployments, 
and expecting a project to own every other piece of their ecosystem is a pretty 
heavy burden. Is there some compromise we can come to? Can we store certain 
pieces such as source code, Dockerfile, and docker binary in some Apache cold 
storage to ensure they are reproduceable?

Ideally I'd just like to keep the burden on project maintainers to a minimum. 
This could heavily discourage projects from creating official helm charts.


On 2020/09/04 15:05:56, Scott Rigby  wrote: 
> Jarek, this is fantastic! Please let me know if I/we can help with this. The 
> Helm team has written tools to make some of these processes easier, including 
> steps to preserve former git history, and GitHub Actions for CI (chart 
> testing) and CD (releasing chart version packages via GitHub release 
> artifacts).
> 
> Bertrand, many orgs are adopting a git repo naming convention "helm-charts" 
> to make this explicit.
> 
> Fun follow-up question: a helm repo is really just an index (YAML) file of 
> metadata in a format the helm client expects, including links to packaged 
> tarballs for each immutable version of a chart. The idea of Apache mirror 
> hosting is on the table, one issue we have is that the GCP object storage 
> buckets that currently host all previous versions of stable and incubator 
> repo charts will be deleted after support for these repos ends on Nov 13th 
> (https://github.com/helm/charts/blob/master/README.md#deprecation-timeline). 
> Would it be possible for Apache to host a mirror of all these packages (note, 
> they are all Apache licenced), or only chart packages related to Apache 
> software (airflow, solr, spark, etc)?
> 
> Scott
> 
> On 2020/09/04 08:47:30, Jarek Potiuk  wrote: 
> > Big +1 for doing this within existing ASF infrastructure. Maybe indeed
> > current SVN  publishing a release snapshot  + mirroring is the best way.
> > 
> > IMHO such 'released' charts do not have to have commit history at all
> > (except 'release' commits) - just release 'snapshots' is enough. For all
> > the other development uses, whatever the source repository is, you will be
> > able too install 'development' chart from the sources.
> > 
> > Still having clear guidelines on how to approach Docker images and
> > 'rebuildability' by the users from sources is an important aspect IMHO.
> > J.
> > 
> > pt., 4 wrz 2020, 10:15 użytkownik Bertrand Delacretaz <
> > bdelacre...@apache.org> napisał:
> > 
> > > Hi,
> > >
> > > On Fri, Sep 4, 2020 at 9:22 AM Jarek Potiuk  wrote:
> > > > ...I think having ASF "apache/chart" for all "officially released" helm
> > > > charts is the best solution
> > >
> > > Sounds like a good idea but please use a more specific name, "charts"
> > > is very generic.
> > >
> > > If distributing those Helm charts does not requires more than a web
> > > server (with some metadata files I suppose) that can probably go in a
> > > specific folder under https://dist.apache.org/repos/dist/release/
> > >
> > > The next question is what kind of traffic is expected there, and can
> > > the clients which will download those Helm charts handle redirects to
> > > distribution mirrors? If yes the mechanism of
> > > http://www.apache.org/dev/release-download-pages.html comes to mind,
> > > if you can piggyback on that mirroring infrastructure that's a big
> > > plus IMO.
> > >
> > > I guess the way to move forward is for the people interested in this
> > > to list the technical and licensing/legal requirements on a wiki page
> > > or equivalent, to discuss with the ASF infrastructure team how to best
> > > support them.
> > >
> > > -Bertrand
> > >
> > > -
> > > To unsubscribe, e-mail: dev-unsubscr...@community.apache.org
> > > For additional commands, e-mail: dev-h...@community.apache.org
> > >
> > >
> > 
> 
> -
> To unsubscribe, e-mail: dev-unsubscr...@community.apache.org
> For additional commands, e-mail: dev-h...@community.apache.org
> 
> 

-
To unsubscribe, e-mail: dev-unsubscr...@community.apache.org
For additional commands, e-mail: dev-h...@community.apache.org

Re: Migration and consolidation helm charts for ASF projects from helm/charts to apache/charts git

[Scott Rigby]

Jarek, this is fantastic! Please let me know if I/we can help with this. The 
Helm team has written tools to make some of these processes easier, including 
steps to preserve former git history, and GitHub Actions for CI (chart testing) 
and CD (releasing chart version packages via GitHub release artifacts).

Bertrand, many orgs are adopting a git repo naming convention "helm-charts" to 
make this explicit.

Fun follow-up question: a helm repo is really just an index (YAML) file of 
metadata in a format the helm client expects, including links to packaged 
tarballs for each immutable version of a chart. The idea of Apache mirror 
hosting is on the table, one issue we have is that the GCP object storage 
buckets that currently host all previous versions of stable and incubator repo 
charts will be deleted after support for these repos ends on Nov 13th 
(https://github.com/helm/charts/blob/master/README.md#deprecation-timeline). 
Would it be possible for Apache to host a mirror of all these packages (note, 
they are all Apache licenced), or only chart packages related to Apache 
software (airflow, solr, spark, etc)?

Scott

On 2020/09/04 08:47:30, Jarek Potiuk  wrote: 
> Big +1 for doing this within existing ASF infrastructure. Maybe indeed
> current SVN  publishing a release snapshot  + mirroring is the best way.
> 
> IMHO such 'released' charts do not have to have commit history at all
> (except 'release' commits) - just release 'snapshots' is enough. For all
> the other development uses, whatever the source repository is, you will be
> able too install 'development' chart from the sources.
> 
> Still having clear guidelines on how to approach Docker images and
> 'rebuildability' by the users from sources is an important aspect IMHO.
> J.
> 
> pt., 4 wrz 2020, 10:15 użytkownik Bertrand Delacretaz <
> bdelacre...@apache.org> napisał:
> 
> > Hi,
> >
> > On Fri, Sep 4, 2020 at 9:22 AM Jarek Potiuk  wrote:
> > > ...I think having ASF "apache/chart" for all "officially released" helm
> > > charts is the best solution
> >
> > Sounds like a good idea but please use a more specific name, "charts"
> > is very generic.
> >
> > If distributing those Helm charts does not requires more than a web
> > server (with some metadata files I suppose) that can probably go in a
> > specific folder under https://dist.apache.org/repos/dist/release/
> >
> > The next question is what kind of traffic is expected there, and can
> > the clients which will download those Helm charts handle redirects to
> > distribution mirrors? If yes the mechanism of
> > http://www.apache.org/dev/release-download-pages.html comes to mind,
> > if you can piggyback on that mirroring infrastructure that's a big
> > plus IMO.
> >
> > I guess the way to move forward is for the people interested in this
> > to list the technical and licensing/legal requirements on a wiki page
> > or equivalent, to discuss with the ASF infrastructure team how to best
> > support them.
> >
> > -Bertrand
> >
> > -
> > To unsubscribe, e-mail: dev-unsubscr...@community.apache.org
> > For additional commands, e-mail: dev-h...@community.apache.org
> >
> >
> 

-
To unsubscribe, e-mail: dev-unsubscr...@community.apache.org
For additional commands, e-mail: dev-h...@community.apache.org

Re: Migration and consolidation helm charts for ASF projects from helm/charts to apache/charts git

[Gaetan Semet]

Hi,

I am one of the maintainer of the Airflow Chart (stable/airflow at 
https://github.com/helm/charts/tree/master/stable/airflow).

Our idea (described in this discussion 
https://github.com/apache/airflow/issues/10523) is to copy or host the current 
Airflow chart as it is, like a "community maintained project" but hosted at ASF 
somewhere. Ideally, we can continue to maintain or maybe even do some evolution 
on it, but the goal is to redirect users to the official Airflow Chart 
(Bitnami?) so new users can select the one they want, and old users can choose 
when to do the migration, even after november).

Gaetan

On 2020/09/03 21:01:12, Dave Fisher  wrote: 
> Hi -
> 
> Does the Helm Chart community wish to explore becoming an Apache Project 
> Community?
> 
> Regards,
> Dave
> 
> Sent from my iPhone
> 
> > On Sep 3, 2020, at 12:33 PM, Ween Jiann  wrote:
> > 
> > Hi all,
> > 
> > The helm team has deprecated the charts repository and will be archiving it 
> > in Nov. Here’s the link to the notice 
> > https://github.com/helm/charts/blob/master/README.md#deprecation-timeline
> > 
> > It looks like that are quite a few charts for ASF projects such as Airflow, 
> > Hadoop, Spark, Solr (not exhaustive) that have yet to be adopted by anyone. 
> > It would make sense to re-home them into a single git in for example 
> > apache/charts. This would facilitate easier distribution of various ASF 
> > projects via a single helm repo, and setting up tests and CI integration 
> > needs only to be done once.
> > 
> > Cheers,
> > WJ
> > 
> > -
> > To unsubscribe, e-mail: dev-unsubscr...@community.apache.org
> > For additional commands, e-mail: dev-h...@community.apache.org
> > 
> 
> 
> -
> To unsubscribe, e-mail: dev-unsubscr...@community.apache.org
> For additional commands, e-mail: dev-h...@community.apache.org
> 
> 

-
To unsubscribe, e-mail: dev-unsubscr...@community.apache.org
For additional commands, e-mail: dev-h...@community.apache.org

Re: Migration and consolidation helm charts for ASF projects from helm/charts to apache/charts git

[Jarek Potiuk]

Big +1 for doing this within existing ASF infrastructure. Maybe indeed
current SVN  publishing a release snapshot  + mirroring is the best way.

IMHO such 'released' charts do not have to have commit history at all
(except 'release' commits) - just release 'snapshots' is enough. For all
the other development uses, whatever the source repository is, you will be
able too install 'development' chart from the sources.

Still having clear guidelines on how to approach Docker images and
'rebuildability' by the users from sources is an important aspect IMHO.
J.

pt., 4 wrz 2020, 10:15 użytkownik Bertrand Delacretaz <
bdelacre...@apache.org> napisał:

> Hi,
>
> On Fri, Sep 4, 2020 at 9:22 AM Jarek Potiuk  wrote:
> > ...I think having ASF "apache/chart" for all "officially released" helm
> > charts is the best solution
>
> Sounds like a good idea but please use a more specific name, "charts"
> is very generic.
>
> If distributing those Helm charts does not requires more than a web
> server (with some metadata files I suppose) that can probably go in a
> specific folder under https://dist.apache.org/repos/dist/release/
>
> The next question is what kind of traffic is expected there, and can
> the clients which will download those Helm charts handle redirects to
> distribution mirrors? If yes the mechanism of
> http://www.apache.org/dev/release-download-pages.html comes to mind,
> if you can piggyback on that mirroring infrastructure that's a big
> plus IMO.
>
> I guess the way to move forward is for the people interested in this
> to list the technical and licensing/legal requirements on a wiki page
> or equivalent, to discuss with the ASF infrastructure team how to best
> support them.
>
> -Bertrand
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@community.apache.org
> For additional commands, e-mail: dev-h...@community.apache.org
>
>

Re: Migration and consolidation helm charts for ASF projects from helm/charts to apache/charts git

[Bertrand Delacretaz]

Hi,

On Fri, Sep 4, 2020 at 9:22 AM Jarek Potiuk  wrote:
> ...I think having ASF "apache/chart" for all "officially released" helm
> charts is the best solution

Sounds like a good idea but please use a more specific name, "charts"
is very generic.

If distributing those Helm charts does not requires more than a web
server (with some metadata files I suppose) that can probably go in a
specific folder under https://dist.apache.org/repos/dist/release/

The next question is what kind of traffic is expected there, and can
the clients which will download those Helm charts handle redirects to
distribution mirrors? If yes the mechanism of
http://www.apache.org/dev/release-download-pages.html comes to mind,
if you can piggyback on that mirroring infrastructure that's a big
plus IMO.

I guess the way to move forward is for the people interested in this
to list the technical and licensing/legal requirements on a wiki page
or equivalent, to discuss with the ASF infrastructure team how to best
support them.

-Bertrand

-
To unsubscribe, e-mail: dev-unsubscr...@community.apache.org
For additional commands, e-mail: dev-h...@community.apache.org

Re: Migration and consolidation helm charts for ASF projects from helm/charts to apache/charts git

[Jarek Potiuk]

I've been thinking a lot and discussing the issues mentioned here (as
in Apache Airflow we had this very case to solve. I would like to extend
what Kamil wrote - and comment on what Apache Airflow uses and some of the
recent discussions we had. I think we had very recently a lot of
discussions about the Helm chart, licencing, releasing mechanisms so
that might be a useful experience for everyone.

TL;DR; I think it is a GREAT idea to have a single place from where all ASF
charts will be hosted. That has a number of advantages and solves a number
of pain points we experienced so far with the distribution mechanism of the
Helm Charts and I think it's a great opportunity to plug a small
"ambiguity" in the ASF release policy - coming from the way how Helm and
Docker Images in general are a bit different than the "existing release
mechanism". This is not a big ambiguity IMHO and the current rules of the
release policy can be interpreted and applied to Helm Chart policies, but
it would be simply great to make clear rules about both - images and charts
as they will likely become a very  important distribution and release
mechanism in the near future (if not already did).

Some ideas/proposals resulting from the discussions I raised:

1) I think having ASF "apache/chart" for all "officially released" helm
charts is the best solution. This should not be a place where the helm
charts are developed - but you should be able to get a released and tagged
versions of those charts. Each project should have their own way of
developing the chart (might be in mono-repo with other parts of the system
or separate repo). For example in Airflow we are developing chart together
with main Airflow code and we are using it during the CI to run tests -
effectively testing "official image", "airflow code" and "airlfow chart"
from the latest master together - so we chose to keep monorepo with all
those. But for other projects, they might have "apache/project-chart" repo
with only the chart. This should be project-based decision.

2) There are already tools (such as https://github.com/google/copybara)
that allow to copy selectively parts of the repositories. And with Github
Actions, we could very easily write an automation where each project could
configure where their chart repository is, which folder, branch and tags
are used and the common "ASF Chart" repo could easily pull-in such charts.
This way releases marked as tags in the original projects would
automatically land in the "ASF Chart" repo - also some validation might be
put in place to make sure the pulled chart is "good to go" and some
notifications might be in place if they are not. I am super-happy to help
with such automation - I've done a lot of Github Actions work recently for
Apache Airflow and helped Apache Beam as well - including developing a very
useful https://github.com/marketplace/actions/cancel-workflow-runs action
that cancels duplicate Github Actions Workflow and helps us to optimise our
Job usage and CI feedback time by 30%.

3) Regarding the licencing, policies - I started recently a discussion
about this very subject
https://lists.apache.org/thread.html/rf2af2a95e7687fe94ede23fe9df388f784c8231a5968b109f677cbe8%40%3Cbuilds.apache.org%3E
and
there was no "definittive" or "clear" answers. There is no direct
mentioning of Helm or Docker release mechanisms in the
http://www.apache.org/legal/release-policy.html but IMHO we can interpret
some of the statements there. I will put relevant parts from the policy
with my interpretation:

a) Docker Images are - rather clearly - binary convenience packages and we
are free to provide them to the users as such - as long we also give users
a way to recreate them as long as they have necessary platform and tools.
So all the sources required to rebuild those images MUST be released as an
"official source package" - following all the rules of ASF (voting, signing
etc.). According to that - I think there is no need to vote on releasing
the Docker Images as long as they are build the same sources that have been
officially voted and released.

> As a convenience to users that might not have the appropriate tools to
build a compiled version of the source, binary/bytecode packages MAY be
distributed alongside official Apache releases. In all such cases, the
binary/bytecode package MUST have the same version number as the source
release and MUST only add binary/bytecode files that are the result of
compiling that version of the source code release and its dependencies.
b) Helm Chart sources - they are sources not binary releases. So sources of
the Helm Chart should be officially voted, signed and released. However
the official release process should be the regular "release process" and
can be made part of the source package in "downloads.apache.org". In case
of Airflow's monorepo for example - we will just include helm chart sources
next time when we release Airflow 1.10.13 and this will also be the
"official" Helm Chart release. We 

Re: Migration and consolidation helm charts for ASF projects from helm/charts to apache/charts git

[Matt Sicker]

If Helm charts have similar licensing complexities as Docker images, then
it probably makes sense to migrate charts to individual projects rather
than making a larger chart repository here at Apache. If they don’t have
similar licensing problems, then it could be aggregated like the Docker
org, though that’s an orthogonal concern at this point based on how Helm
charts have migrated from the centralized distribution model. Helm makes it
fairly easy to add chart repositories (at least in v3) so that individual
projects can own their own charts.

On Thu, Sep 3, 2020 at 22:32 Dave Fisher  wrote:

> The ASF and the Apache Way is all about the community around an open
> source software project. If there is no community around this deprecated
> package then you will need to contact each Apache community that might be
> interested and ask them directly.
>
>
>
> If you have a group of developers who want to preserve this project then
> incubating an Apache project could be an approach.
>
>
>
> Regards,
>
> Dave
>
>
>
> Sent from my iPhone
>
>
>
> > On Sep 3, 2020, at 7:00 PM, Ween Jiann  wrote:
>
> >
>
> > Hi Dave,
>
> >
>
> > Just to clarify, my suggestion is to move only charts related to ASF
> projects to the apache git repo.
>
> > These charts have been deprecated as part of the repo deprecation, they
> have no choice but to move somewhere or be archived.
>
> >
>
> > If this gains some traction, I'll create an issue and contact the
> maintainers of the charts.
>
> >
>
> >> On 2020/09/03 21:01:12, Dave Fisher  wrote:
>
> >> Hi -
>
> >>
>
> >> Does the Helm Chart community wish to explore becoming an Apache
> Project Community?
>
> >>
>
> >> Regards,
>
> >> Dave
>
> >>
>
> >> Sent from my iPhone
>
> >>
>
>  On Sep 3, 2020, at 12:33 PM, Ween Jiann 
> wrote:
>
> >>>
>
> >>> Hi all,
>
> >>>
>
> >>> The helm team has deprecated the charts repository and will be
> archiving it in Nov. Here’s the link to the notice
> https://github.com/helm/charts/blob/master/README.md#deprecation-timeline
>
> >>>
>
> >>> It looks like that are quite a few charts for ASF projects such as
> Airflow, Hadoop, Spark, Solr (not exhaustive) that have yet to be adopted
> by anyone. It would make sense to re-home them into a single git in for
> example apache/charts. This would facilitate easier distribution of various
> ASF projects via a single helm repo, and setting up tests and CI
> integration needs only to be done once.
>
> >>>
>
> >>> Cheers,
>
> >>> WJ
>
> >>>
>
> >>> -
>
> >>> To unsubscribe, e-mail: dev-unsubscr...@community.apache.org
>
> >>> For additional commands, e-mail: dev-h...@community.apache.org
>
> >>>
>
> >>
>
> >>
>
> >> -
>
> >> To unsubscribe, e-mail: dev-unsubscr...@community.apache.org
>
> >> For additional commands, e-mail: dev-h...@community.apache.org
>
> >>
>
> >>
>
> >
>
> > -
>
> > To unsubscribe, e-mail: dev-unsubscr...@community.apache.org
>
> > For additional commands, e-mail: dev-h...@community.apache.org
>
> >
>
>
>
>
>
> -
>
> To unsubscribe, e-mail: dev-unsubscr...@community.apache.org
>
> For additional commands, e-mail: dev-h...@community.apache.org
>
>
>
> --
Matt Sicker 

Re: Migration and consolidation helm charts for ASF projects from helm/charts to apache/charts git

[Dave Fisher]

The ASF and the Apache Way is all about the community around an open source 
software project. If there is no community around this deprecated package then 
you will need to contact each Apache community that might be interested and ask 
them directly.

If you have a group of developers who want to preserve this project then 
incubating an Apache project could be an approach.

Regards,
Dave

Sent from my iPhone

> On Sep 3, 2020, at 7:00 PM, Ween Jiann  wrote:
> 
> Hi Dave,
> 
> Just to clarify, my suggestion is to move only charts related to ASF projects 
> to the apache git repo.
> These charts have been deprecated as part of the repo deprecation, they have 
> no choice but to move somewhere or be archived.
> 
> If this gains some traction, I'll create an issue and contact the maintainers 
> of the charts.
> 
>> On 2020/09/03 21:01:12, Dave Fisher  wrote: 
>> Hi -
>> 
>> Does the Helm Chart community wish to explore becoming an Apache Project 
>> Community?
>> 
>> Regards,
>> Dave
>> 
>> Sent from my iPhone
>> 
 On Sep 3, 2020, at 12:33 PM, Ween Jiann  wrote:
>>> 
>>> Hi all,
>>> 
>>> The helm team has deprecated the charts repository and will be archiving it 
>>> in Nov. Here’s the link to the notice 
>>> https://github.com/helm/charts/blob/master/README.md#deprecation-timeline
>>> 
>>> It looks like that are quite a few charts for ASF projects such as Airflow, 
>>> Hadoop, Spark, Solr (not exhaustive) that have yet to be adopted by anyone. 
>>> It would make sense to re-home them into a single git in for example 
>>> apache/charts. This would facilitate easier distribution of various ASF 
>>> projects via a single helm repo, and setting up tests and CI integration 
>>> needs only to be done once.
>>> 
>>> Cheers,
>>> WJ
>>> 
>>> -
>>> To unsubscribe, e-mail: dev-unsubscr...@community.apache.org
>>> For additional commands, e-mail: dev-h...@community.apache.org
>>> 
>> 
>> 
>> -
>> To unsubscribe, e-mail: dev-unsubscr...@community.apache.org
>> For additional commands, e-mail: dev-h...@community.apache.org
>> 
>> 
> 
> -
> To unsubscribe, e-mail: dev-unsubscr...@community.apache.org
> For additional commands, e-mail: dev-h...@community.apache.org
> 


-
To unsubscribe, e-mail: dev-unsubscr...@community.apache.org
For additional commands, e-mail: dev-h...@community.apache.org

Re: Migration and consolidation helm charts for ASF projects from helm/charts to apache/charts git

[Ween Jiann]

Hi Kamil,

What's your take of having the helm chart in a single repo?
It will lessen the load to maintain tests and distribution.

On 2020/09/03 19:54:17, Kamil Breguła  wrote: 
> Hello,
> 
> In the case of Airflow, we have developed our own chart and want to develop
> and release it soon. For now, there is little chance that our community
> will be interested in developing two independent charts.
> https://github.com/apache/airflow/issues/10523
> 
> That would be the 3rd method of deploying Airflow on Kubernetes maintained
> by our project. We have Kubernetes Operators also.
> https://github.com/apache/airflow-on-k8s-operator
> 
> Best regards,
> Kamil Breguła
> 
> On Thu, Sep 3, 2020 at 9:33 PM Ween Jiann  wrote:
> 
> > Hi all,
> >
> > The helm team has deprecated the charts repository and will be archiving
> > it in Nov. Here’s the link to the notice
> > https://github.com/helm/charts/blob/master/README.md#deprecation-timeline
> >
> > It looks like that are quite a few charts for ASF projects such as
> > Airflow, Hadoop, Spark, Solr (not exhaustive) that have yet to be adopted
> > by anyone. It would make sense to re-home them into a single git in for
> > example apache/charts. This would facilitate easier distribution of various
> > ASF projects via a single helm repo, and setting up tests and CI
> > integration needs only to be done once.
> >
> > Cheers,
> > WJ
> >
> > -
> > To unsubscribe, e-mail: dev-unsubscr...@community.apache.org
> > For additional commands, e-mail: dev-h...@community.apache.org
> >
> >
> 

-
To unsubscribe, e-mail: dev-unsubscr...@community.apache.org
For additional commands, e-mail: dev-h...@community.apache.org

Re: Migration and consolidation helm charts for ASF projects from helm/charts to apache/charts git

[Ween Jiann]

Hi Dave,

Just to clarify, my suggestion is to move only charts related to ASF projects 
to the apache git repo.
These charts have been deprecated as part of the repo deprecation, they have no 
choice but to move somewhere or be archived.

If this gains some traction, I'll create an issue and contact the maintainers 
of the charts.

On 2020/09/03 21:01:12, Dave Fisher  wrote: 
> Hi -
> 
> Does the Helm Chart community wish to explore becoming an Apache Project 
> Community?
> 
> Regards,
> Dave
> 
> Sent from my iPhone
> 
> > On Sep 3, 2020, at 12:33 PM, Ween Jiann  wrote:
> > 
> > Hi all,
> > 
> > The helm team has deprecated the charts repository and will be archiving it 
> > in Nov. Here’s the link to the notice 
> > https://github.com/helm/charts/blob/master/README.md#deprecation-timeline
> > 
> > It looks like that are quite a few charts for ASF projects such as Airflow, 
> > Hadoop, Spark, Solr (not exhaustive) that have yet to be adopted by anyone. 
> > It would make sense to re-home them into a single git in for example 
> > apache/charts. This would facilitate easier distribution of various ASF 
> > projects via a single helm repo, and setting up tests and CI integration 
> > needs only to be done once.
> > 
> > Cheers,
> > WJ
> > 
> > -
> > To unsubscribe, e-mail: dev-unsubscr...@community.apache.org
> > For additional commands, e-mail: dev-h...@community.apache.org
> > 
> 
> 
> -
> To unsubscribe, e-mail: dev-unsubscr...@community.apache.org
> For additional commands, e-mail: dev-h...@community.apache.org
> 
> 

-
To unsubscribe, e-mail: dev-unsubscr...@community.apache.org
For additional commands, e-mail: dev-h...@community.apache.org

Re: Migration and consolidation helm charts for ASF projects from helm/charts to apache/charts git

[Dave Fisher]

Hi -

Does the Helm Chart community wish to explore becoming an Apache Project 
Community?

Regards,
Dave

Sent from my iPhone

> On Sep 3, 2020, at 12:33 PM, Ween Jiann  wrote:
> 
> Hi all,
> 
> The helm team has deprecated the charts repository and will be archiving it 
> in Nov. Here’s the link to the notice 
> https://github.com/helm/charts/blob/master/README.md#deprecation-timeline
> 
> It looks like that are quite a few charts for ASF projects such as Airflow, 
> Hadoop, Spark, Solr (not exhaustive) that have yet to be adopted by anyone. 
> It would make sense to re-home them into a single git in for example 
> apache/charts. This would facilitate easier distribution of various ASF 
> projects via a single helm repo, and setting up tests and CI integration 
> needs only to be done once.
> 
> Cheers,
> WJ
> 
> -
> To unsubscribe, e-mail: dev-unsubscr...@community.apache.org
> For additional commands, e-mail: dev-h...@community.apache.org
> 


-
To unsubscribe, e-mail: dev-unsubscr...@community.apache.org
For additional commands, e-mail: dev-h...@community.apache.org

Re: Migration and consolidation helm charts for ASF projects from helm/charts to apache/charts git

[Kamil Breguła]

Hello,

In the case of Airflow, we have developed our own chart and want to develop
and release it soon. For now, there is little chance that our community
will be interested in developing two independent charts.
https://github.com/apache/airflow/issues/10523

That would be the 3rd method of deploying Airflow on Kubernetes maintained
by our project. We have Kubernetes Operators also.
https://github.com/apache/airflow-on-k8s-operator

Best regards,
Kamil Breguła

On Thu, Sep 3, 2020 at 9:33 PM Ween Jiann  wrote:

> Hi all,
>
> The helm team has deprecated the charts repository and will be archiving
> it in Nov. Here’s the link to the notice
> https://github.com/helm/charts/blob/master/README.md#deprecation-timeline
>
> It looks like that are quite a few charts for ASF projects such as
> Airflow, Hadoop, Spark, Solr (not exhaustive) that have yet to be adopted
> by anyone. It would make sense to re-home them into a single git in for
> example apache/charts. This would facilitate easier distribution of various
> ASF projects via a single helm repo, and setting up tests and CI
> integration needs only to be done once.
>
> Cheers,
> WJ
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@community.apache.org
> For additional commands, e-mail: dev-h...@community.apache.org
>
>

Migration and consolidation helm charts for ASF projects from helm/charts to apache/charts git

[Ween Jiann]

Hi all,
 
The helm team has deprecated the charts repository and will be archiving it in 
Nov. Here’s the link to the notice 
https://github.com/helm/charts/blob/master/README.md#deprecation-timeline
 
It looks like that are quite a few charts for ASF projects such as Airflow, 
Hadoop, Spark, Solr (not exhaustive) that have yet to be adopted by anyone. It 
would make sense to re-home them into a single git in for example 
apache/charts. This would facilitate easier distribution of various ASF 
projects via a single helm repo, and setting up tests and CI integration needs 
only to be done once.
 
Cheers,
WJ

-
To unsubscribe, e-mail: dev-unsubscr...@community.apache.org
For additional commands, e-mail: dev-h...@community.apache.org