Hey Dave, Thanks for this. Having the ComDev/Infra repository where PMC can publish approved "Chart" and clear docs on what it means to follow the Release Policy is exactly what we need I think.
We have a heated debate within Airflow PMC and despite exchanging many messages and (sometimes heated) arguments we remain deeply divided on that. For me, that's a clear sign that we need the ASF help to resolve it. How can I help to make it happen? I am happy to devote quite a lot of my time - both infrastructure and policy discussions/writing as I think this is a very important subject to at least some commercial users of ASF software (and ASF being 'commercially friendly'). J. On Sun, Sep 6, 2020 at 1:19 AM Kaxil Naik <kaxiln...@gmail.com> wrote: > Great, thanks Dave > > On Sun, Sep 6, 2020, 00:17 Dave Fisher <wave4d...@comcast.net> wrote: > > > Every release must be made in SVN on dist.apache.org. All other channels > > are optional and may or may not have Infra support. > > > > I view this as a request to have ComDev or Infra manage a Gitbox/Github > > repository where any PMC can publish a PMC approved Helm Chart release > from > > their project. > > > > How to make a Helm Chart compliant with Apache Release Policy would part > > of the documentation. > > > > Sent from my iPhone > > > > > On Sep 5, 2020, at 3:30 PM, Kaxil Naik <kaxiln...@gmail.com> wrote: > > > > > > > > >> > > >> The point is that we MUST give the users possibility to (re)build > those > > > images on their own if they want. We > > > > > > > > > From http://www.apache.org/legal/release-policy.html#source-packages, > it > > > does not say it has to be on PyPI. I will reiterate what I said in the > > > Github issue, having a binary on PyPI is no guarantee that it will stay > > > there. Having a binary on Github releases > > > https://github.com/jbub/pgbouncer_exporter/tags also does not > guarantee > > it > > > would not change. If the license are correct and the source is > available > > I > > > definitely do not see this as a problem: > > > > > > Every ASF release MUST contain one or more source packages, which MUST > be > > >> sufficient for a user to build and test the release provided they have > > >> access to the appropriate platform and tools. > > > > > > > > > > > >> On Sat, Sep 5, 2020 at 10:33 PM Jarek Potiuk <ja...@potiuk.com> > wrote: > > >> > > >> @Matt Sicker <boa...@gmail.com> -> agree on both, central chart repo, > > and > > >> per-project sources for charts. Though packaging is a bit different > than > > >> Docker though. The packages contain tar.gz sources of the chart (which > > >> usually includes quite complex go templating logic, not just simple > yaml > > >> files) and those templates include names of the images used. This > makes > > >> the Helm chart much more close to the -src.tar.gz files we release now > > than > > >> to Docker Images we publish. So IMHO the .tgz part should follow the > > normal > > >> release process with voting @Kaxilnaik <kaxiln...@apache.org>. > > >> Unless those are the very same sources of product that are already > > formally > > >> released. But then @Gaetan Semet previously pointed out as bad > practice > > >> to bind the two together. > > >> IMHO that means that we have to vote on each Chart release. But I'd > love > > >> what others think - knowing this packaging details. > > >> > > >> @ermanndimb- my point for images pointed to by the helm chart is not > > about > > >> copying full sources (I think from this discussion I realised where > the > > >> main confusion was) . This is not needed. I just made a PR to show how > > this > > >> can be done with the pgbouncer-exporter. This is quite a good example > > to > > >> base our discussion on: https://github.com/apache/airflow/pull/10759 > > >> > > >> The point is that we MUST give the users possibility to (re)build > those > > >> images on their own if they want. They should be able to do that > using: > > >> > > >> a) official base images (DockerHub has the "official image program" > for > > >> platforms. openjdk, python,debian, alpine - those are all official > > images > > >> in DockerHub terms > > >> b) having access to properly licensed source code with some > guarantees > > >> that it will not disappear ("official" repo, multiple forks in Github, > > >> heavily used by others, fork under "apache" organisation) . > > >> c) instructions where to find the sources and build the images > > >> (Dockerfile + necessary scripts to build the image if in case it is > not > > >> straightforward). Similarly as we provide instructions on building the > > >> "product". > > >> > > >> If the first two are available but it's not obvious how the image was > > built > > >> - IMHO we need to provide the user instructions on how to build those > > >> images (ideally a script that does it automatically). > > >> > > >> This is all fulfilled by the PR above as an example: > > >> a) based on official, alpine image > > >> b) the pgbouncer code is properly licenced and we forked the sources > > >> several times to be sure it won't disappear > > >> c) we have a script that builds and publishes the image (image is > > published > > >> under apache/airflow DockerHub) > > >> > > >> IMHO - that's a very good example to show the kind of approach we can > > take, > > >> and shows that it isn't difficult to handle it well at all. > > >> > > >> The alternative that I am afraid of is - that we require our users to > > use > > >> binaries which are not "official" but maintained and controlled by > > >> 3rd-party (without a straightforward way of building the binary from > > >> official images + properly licensed sources + instructions). If we do > > not > > >> give those to the user but we simply point to a 3rd-party binary > bimage > > - > > >> we both endorse the 3rd party, and introduce dependency on the > > 3rd-party. > > >> The user then has to rely on the 3rd-party to use the Airflow Helm > > Chart - > > >> which I think is a very bad idea. > > >> > > >> J. > > >> > > >> > > >>> On Sat, Sep 5, 2020 at 7:42 PM Kaxil Naik <kaxiln...@gmail.com> > wrote: > > >>> > > >>> Does voting needs to happen for "releasing" the Helm chart ? Do we > any > > >>> guidelines from the ASF around releasing Helm Chart & Dockerfile? > > >>> > > >>>> On Sat, Sep 5, 2020, 17:36 Matt Sicker <boa...@gmail.com> wrote: > > >>> > > >>>> If packaging is similar to Docker, then our Helm charts will still > > >>>> need some third party base images and such for GPL licensed system > > >>>> dependencies (besides the kernel itself, there's OpenJDK which is > > >>>> definitely used in plenty of projects here). The bits we build on > top > > >>>> of external components becomes the Apache project that is ALv2 and > is > > >>>> distributed through source archives with convenience binaries (in > this > > >>>> case, the image layers and YAML files). > > >>>> > > >>>> I do like the idea of having a central Apache Helm chart repo where > > >>>> releases can be published. That makes it simpler for users to > combine > > >>>> multiple Apache charts together, too, based on my past experience > with > > >>>> using Helm (especially with the new decentralized charts > distribution > > >>>> model they've switched to). I do think it makes sense to try and > > >>>> maintain the sources and such for those charts in the individual > > >>>> projects since each PMC may have different workflows on how that's > > >>>> created, and it makes it easier to release alongside the normal > > >>>> project's releases. > > >>>> > > >>>> On Sat, 5 Sep 2020 at 11:13, Daniel Imberman <dimber...@apache.org> > > >>> wrote: > > >>>>> > > >>>>> While I understand the necessity for users to have access to all > > >> source > > >>>> code, I'm a bit concerned about the complexity this places on the > > >>>> open-source project. Let's take Airflow as an example. If we want to > > >> use > > >>>> pgbouncer in our home chart are we expected to be able to build this > > >>>> external project? > > >>>>> > > >>>>> I think that if this is ASF policy, then maybe we need to modify > the > > >>>> policy to meet the current ecosystem. Helm charts are made to handle > > >>>> complex deployments, and expecting a project to own every other > piece > > >> of > > >>>> their ecosystem is a pretty heavy burden. Is there some compromise > we > > >> can > > >>>> come to? Can we store certain pieces such as source code, > Dockerfile, > > >> and > > >>>> docker binary in some Apache cold storage to ensure they are > > >>> reproduceable? > > >>>>> > > >>>>> Ideally I'd just like to keep the burden on project maintainers to > a > > >>>> minimum. This could heavily discourage projects from creating > official > > >>> helm > > >>>> charts. > > >>>>> > > >>>>> > > >>>>> On 2020/09/04 15:05:56, Scott Rigby <sc...@r6by.com> wrote: > > >>>>>> Jarek, this is fantastic! Please let me know if I/we can help with > > >>>> this. The Helm team has written tools to make some of these > processes > > >>>> easier, including steps to preserve former git history, and GitHub > > >>> Actions > > >>>> for CI (chart testing) and CD (releasing chart version packages via > > >>> GitHub > > >>>> release artifacts). > > >>>>>> > > >>>>>> Bertrand, many orgs are adopting a git repo naming convention > > >>>> "helm-charts" to make this explicit. > > >>>>>> > > >>>>>> Fun follow-up question: a helm repo is really just an index (YAML) > > >>>> file of metadata in a format the helm client expects, including > links > > >> to > > >>>> packaged tarballs for each immutable version of a chart. The idea of > > >>> Apache > > >>>> mirror hosting is on the table, one issue we have is that the GCP > > >> object > > >>>> storage buckets that currently host all previous versions of stable > > and > > >>>> incubator repo charts will be deleted after support for these repos > > >> ends > > >>> on > > >>>> Nov 13th ( > > >>>> > > >>> > > >> > > > https://github.com/helm/charts/blob/master/README.md#deprecation-timeline > > >>> ). > > >>>> Would it be possible for Apache to host a mirror of all these > packages > > >>>> (note, they are all Apache licenced), or only chart packages related > > to > > >>>> Apache software (airflow, solr, spark, etc)? > > >>>>>> > > >>>>>> Scott > > >>>>>> > > >>>>>> On 2020/09/04 08:47:30, Jarek Potiuk <ja...@potiuk.com> wrote: > > >>>>>>> Big +1 for doing this within existing ASF infrastructure. Maybe > > >>>> indeed > > >>>>>>> current SVN publishing a release snapshot + mirroring is the > > >> best > > >>>> way. > > >>>>>>> > > >>>>>>> IMHO such 'released' charts do not have to have commit history at > > >>> all > > >>>>>>> (except 'release' commits) - just release 'snapshots' is enough. > > >>> For > > >>>> all > > >>>>>>> the other development uses, whatever the source repository is, > > >> you > > >>>> will be > > >>>>>>> able too install 'development' chart from the sources. > > >>>>>>> > > >>>>>>> Still having clear guidelines on how to approach Docker images > > >> and > > >>>>>>> 'rebuildability' by the users from sources is an important aspect > > >>>> IMHO. > > >>>>>>> J. > > >>>>>>> > > >>>>>>> pt., 4 wrz 2020, 10:15 użytkownik Bertrand Delacretaz < > > >>>>>>> bdelacre...@apache.org> napisał: > > >>>>>>> > > >>>>>>>> Hi, > > >>>>>>>> > > >>>>>>>> On Fri, Sep 4, 2020 at 9:22 AM Jarek Potiuk <ja...@potiuk.com> > > >>>> wrote: > > >>>>>>>>> ...I think having ASF "apache/chart" for all "officially > > >>>> released" helm > > >>>>>>>>> charts is the best solution.... > > >>>>>>>> > > >>>>>>>> Sounds like a good idea but please use a more specific name, > > >>>> "charts" > > >>>>>>>> is very generic. > > >>>>>>>> > > >>>>>>>> If distributing those Helm charts does not requires more than a > > >>> web > > >>>>>>>> server (with some metadata files I suppose) that can probably > > >> go > > >>>> in a > > >>>>>>>> specific folder under > > >>> https://dist.apache.org/repos/dist/release/ > > >>>>>>>> > > >>>>>>>> The next question is what kind of traffic is expected there, > > >> and > > >>>> can > > >>>>>>>> the clients which will download those Helm charts handle > > >>> redirects > > >>>> to > > >>>>>>>> distribution mirrors? If yes the mechanism of > > >>>>>>>> http://www.apache.org/dev/release-download-pages.html comes to > > >>>> mind, > > >>>>>>>> if you can piggyback on that mirroring infrastructure that's a > > >>> big > > >>>>>>>> plus IMO. > > >>>>>>>> > > >>>>>>>> I guess the way to move forward is for the people interested in > > >>>> this > > >>>>>>>> to list the technical and licensing/legal requirements on a > > >> wiki > > >>>> page > > >>>>>>>> or equivalent, to discuss with the ASF infrastructure team how > > >> to > > >>>> best > > >>>>>>>> support them. > > >>>>>>>> > > >>>>>>>> -Bertrand > > >>>>>>>> > > >>>>>>>> > > >>>> > --------------------------------------------------------------------- > > >>>>>>>> To unsubscribe, e-mail: dev-unsubscr...@community.apache.org > > >>>>>>>> For additional commands, e-mail: dev-h...@community.apache.org > > >>>>>>>> > > >>>>>>>> > > >>>>>>> > > >>>>>> > > >>>>>> > > >> --------------------------------------------------------------------- > > >>>>>> To unsubscribe, e-mail: dev-unsubscr...@community.apache.org > > >>>>>> For additional commands, e-mail: dev-h...@community.apache.org > > >>>>>> > > >>>>>> > > >>>>> > > >>>>> > --------------------------------------------------------------------- > > >>>>> To unsubscribe, e-mail: dev-unsubscr...@community.apache.org > > >>>>> For additional commands, e-mail: dev-h...@community.apache.org > > >>>>> > > >>>> > > >>>> > > >>>> -- > > >>>> Matt Sicker <boa...@gmail.com> > > >>>> > > >>>> > --------------------------------------------------------------------- > > >>>> To unsubscribe, e-mail: dev-unsubscr...@community.apache.org > > >>>> For additional commands, e-mail: dev-h...@community.apache.org > > >>>> > > >>>> > > >>> > > >> > > >> > > >> -- > > >> +48 660 796 129 > > >> > > > > > -- +48 660 796 129
