I will be happy to help too, definitely and important topic where we need a clear guidance on it (what sources and binaries are allowed etc)
Regards, Kaxil Airflow PMC On Sun, Sep 6, 2020, 09:37 Jarek Potiuk <ja...@potiuk.com> wrote: > Hey Dave, > > Thanks for this. Having the ComDev/Infra repository where PMC can publish > approved "Chart" and clear docs on what it means to follow the Release > Policy is exactly what we need I think. > > We have a heated debate within Airflow PMC and despite exchanging many > messages and (sometimes heated) arguments we remain deeply divided on that. > For me, that's a clear sign that we need the ASF help to resolve it. > > How can I help to make it happen? > > I am happy to devote quite a lot of my time - both infrastructure and > policy discussions/writing as I think this is a very important subject to > at least some commercial users of ASF software (and ASF being 'commercially > friendly'). > > J. > > > > On Sun, Sep 6, 2020 at 1:19 AM Kaxil Naik <kaxiln...@gmail.com> wrote: > > > Great, thanks Dave > > > > On Sun, Sep 6, 2020, 00:17 Dave Fisher <wave4d...@comcast.net> wrote: > > > > > Every release must be made in SVN on dist.apache.org. All other > channels > > > are optional and may or may not have Infra support. > > > > > > I view this as a request to have ComDev or Infra manage a Gitbox/Github > > > repository where any PMC can publish a PMC approved Helm Chart release > > from > > > their project. > > > > > > How to make a Helm Chart compliant with Apache Release Policy would > part > > > of the documentation. > > > > > > Sent from my iPhone > > > > > > > On Sep 5, 2020, at 3:30 PM, Kaxil Naik <kaxiln...@gmail.com> wrote: > > > > > > > > > > > >> > > > >> The point is that we MUST give the users possibility to (re)build > > those > > > > images on their own if they want. We > > > > > > > > > > > > From http://www.apache.org/legal/release-policy.html#source-packages > , > > it > > > > does not say it has to be on PyPI. I will reiterate what I said in > the > > > > Github issue, having a binary on PyPI is no guarantee that it will > stay > > > > there. Having a binary on Github releases > > > > https://github.com/jbub/pgbouncer_exporter/tags also does not > > guarantee > > > it > > > > would not change. If the license are correct and the source is > > available > > > I > > > > definitely do not see this as a problem: > > > > > > > > Every ASF release MUST contain one or more source packages, which > MUST > > be > > > >> sufficient for a user to build and test the release provided they > have > > > >> access to the appropriate platform and tools. > > > > > > > > > > > > > > > >> On Sat, Sep 5, 2020 at 10:33 PM Jarek Potiuk <ja...@potiuk.com> > > wrote: > > > >> > > > >> @Matt Sicker <boa...@gmail.com> -> agree on both, central chart > repo, > > > and > > > >> per-project sources for charts. Though packaging is a bit different > > than > > > >> Docker though. The packages contain tar.gz sources of the chart > (which > > > >> usually includes quite complex go templating logic, not just simple > > yaml > > > >> files) and those templates include names of the images used. This > > makes > > > >> the Helm chart much more close to the -src.tar.gz files we release > now > > > than > > > >> to Docker Images we publish. So IMHO the .tgz part should follow the > > > normal > > > >> release process with voting @Kaxilnaik <kaxiln...@apache.org>. > > > >> Unless those are the very same sources of product that are already > > > formally > > > >> released. But then @Gaetan Semet previously pointed out as bad > > practice > > > >> to bind the two together. > > > >> IMHO that means that we have to vote on each Chart release. But I'd > > love > > > >> what others think - knowing this packaging details. > > > >> > > > >> @ermanndimb- my point for images pointed to by the helm chart is not > > > about > > > >> copying full sources (I think from this discussion I realised where > > the > > > >> main confusion was) . This is not needed. I just made a PR to show > how > > > this > > > >> can be done with the pgbouncer-exporter. This is quite a good > example > > > to > > > >> base our discussion on: > https://github.com/apache/airflow/pull/10759 > > > >> > > > >> The point is that we MUST give the users possibility to (re)build > > those > > > >> images on their own if they want. They should be able to do that > > using: > > > >> > > > >> a) official base images (DockerHub has the "official image program" > > for > > > >> platforms. openjdk, python,debian, alpine - those are all official > > > images > > > >> in DockerHub terms > > > >> b) having access to properly licensed source code with some > > guarantees > > > >> that it will not disappear ("official" repo, multiple forks in > Github, > > > >> heavily used by others, fork under "apache" organisation) . > > > >> c) instructions where to find the sources and build the images > > > >> (Dockerfile + necessary scripts to build the image if in case it is > > not > > > >> straightforward). Similarly as we provide instructions on building > the > > > >> "product". > > > >> > > > >> If the first two are available but it's not obvious how the image > was > > > built > > > >> - IMHO we need to provide the user instructions on how to build > those > > > >> images (ideally a script that does it automatically). > > > >> > > > >> This is all fulfilled by the PR above as an example: > > > >> a) based on official, alpine image > > > >> b) the pgbouncer code is properly licenced and we forked the sources > > > >> several times to be sure it won't disappear > > > >> c) we have a script that builds and publishes the image (image is > > > published > > > >> under apache/airflow DockerHub) > > > >> > > > >> IMHO - that's a very good example to show the kind of approach we > can > > > take, > > > >> and shows that it isn't difficult to handle it well at all. > > > >> > > > >> The alternative that I am afraid of is - that we require our users > to > > > use > > > >> binaries which are not "official" but maintained and controlled by > > > >> 3rd-party (without a straightforward way of building the binary from > > > >> official images + properly licensed sources + instructions). If we > do > > > not > > > >> give those to the user but we simply point to a 3rd-party binary > > bimage > > > - > > > >> we both endorse the 3rd party, and introduce dependency on the > > > 3rd-party. > > > >> The user then has to rely on the 3rd-party to use the Airflow Helm > > > Chart - > > > >> which I think is a very bad idea. > > > >> > > > >> J. > > > >> > > > >> > > > >>> On Sat, Sep 5, 2020 at 7:42 PM Kaxil Naik <kaxiln...@gmail.com> > > wrote: > > > >>> > > > >>> Does voting needs to happen for "releasing" the Helm chart ? Do we > > any > > > >>> guidelines from the ASF around releasing Helm Chart & Dockerfile? > > > >>> > > > >>>> On Sat, Sep 5, 2020, 17:36 Matt Sicker <boa...@gmail.com> wrote: > > > >>> > > > >>>> If packaging is similar to Docker, then our Helm charts will still > > > >>>> need some third party base images and such for GPL licensed system > > > >>>> dependencies (besides the kernel itself, there's OpenJDK which is > > > >>>> definitely used in plenty of projects here). The bits we build on > > top > > > >>>> of external components becomes the Apache project that is ALv2 and > > is > > > >>>> distributed through source archives with convenience binaries (in > > this > > > >>>> case, the image layers and YAML files). > > > >>>> > > > >>>> I do like the idea of having a central Apache Helm chart repo > where > > > >>>> releases can be published. That makes it simpler for users to > > combine > > > >>>> multiple Apache charts together, too, based on my past experience > > with > > > >>>> using Helm (especially with the new decentralized charts > > distribution > > > >>>> model they've switched to). I do think it makes sense to try and > > > >>>> maintain the sources and such for those charts in the individual > > > >>>> projects since each PMC may have different workflows on how that's > > > >>>> created, and it makes it easier to release alongside the normal > > > >>>> project's releases. > > > >>>> > > > >>>> On Sat, 5 Sep 2020 at 11:13, Daniel Imberman < > dimber...@apache.org> > > > >>> wrote: > > > >>>>> > > > >>>>> While I understand the necessity for users to have access to all > > > >> source > > > >>>> code, I'm a bit concerned about the complexity this places on the > > > >>>> open-source project. Let's take Airflow as an example. If we want > to > > > >> use > > > >>>> pgbouncer in our home chart are we expected to be able to build > this > > > >>>> external project? > > > >>>>> > > > >>>>> I think that if this is ASF policy, then maybe we need to modify > > the > > > >>>> policy to meet the current ecosystem. Helm charts are made to > handle > > > >>>> complex deployments, and expecting a project to own every other > > piece > > > >> of > > > >>>> their ecosystem is a pretty heavy burden. Is there some compromise > > we > > > >> can > > > >>>> come to? Can we store certain pieces such as source code, > > Dockerfile, > > > >> and > > > >>>> docker binary in some Apache cold storage to ensure they are > > > >>> reproduceable? > > > >>>>> > > > >>>>> Ideally I'd just like to keep the burden on project maintainers > to > > a > > > >>>> minimum. This could heavily discourage projects from creating > > official > > > >>> helm > > > >>>> charts. > > > >>>>> > > > >>>>> > > > >>>>> On 2020/09/04 15:05:56, Scott Rigby <sc...@r6by.com> wrote: > > > >>>>>> Jarek, this is fantastic! Please let me know if I/we can help > with > > > >>>> this. The Helm team has written tools to make some of these > > processes > > > >>>> easier, including steps to preserve former git history, and GitHub > > > >>> Actions > > > >>>> for CI (chart testing) and CD (releasing chart version packages > via > > > >>> GitHub > > > >>>> release artifacts). > > > >>>>>> > > > >>>>>> Bertrand, many orgs are adopting a git repo naming convention > > > >>>> "helm-charts" to make this explicit. > > > >>>>>> > > > >>>>>> Fun follow-up question: a helm repo is really just an index > (YAML) > > > >>>> file of metadata in a format the helm client expects, including > > links > > > >> to > > > >>>> packaged tarballs for each immutable version of a chart. The idea > of > > > >>> Apache > > > >>>> mirror hosting is on the table, one issue we have is that the GCP > > > >> object > > > >>>> storage buckets that currently host all previous versions of > stable > > > and > > > >>>> incubator repo charts will be deleted after support for these > repos > > > >> ends > > > >>> on > > > >>>> Nov 13th ( > > > >>>> > > > >>> > > > >> > > > > > > https://github.com/helm/charts/blob/master/README.md#deprecation-timeline > > > >>> ). > > > >>>> Would it be possible for Apache to host a mirror of all these > > packages > > > >>>> (note, they are all Apache licenced), or only chart packages > related > > > to > > > >>>> Apache software (airflow, solr, spark, etc)? > > > >>>>>> > > > >>>>>> Scott > > > >>>>>> > > > >>>>>> On 2020/09/04 08:47:30, Jarek Potiuk <ja...@potiuk.com> wrote: > > > >>>>>>> Big +1 for doing this within existing ASF infrastructure. Maybe > > > >>>> indeed > > > >>>>>>> current SVN publishing a release snapshot + mirroring is the > > > >> best > > > >>>> way. > > > >>>>>>> > > > >>>>>>> IMHO such 'released' charts do not have to have commit history > at > > > >>> all > > > >>>>>>> (except 'release' commits) - just release 'snapshots' is > enough. > > > >>> For > > > >>>> all > > > >>>>>>> the other development uses, whatever the source repository is, > > > >> you > > > >>>> will be > > > >>>>>>> able too install 'development' chart from the sources. > > > >>>>>>> > > > >>>>>>> Still having clear guidelines on how to approach Docker images > > > >> and > > > >>>>>>> 'rebuildability' by the users from sources is an important > aspect > > > >>>> IMHO. > > > >>>>>>> J. > > > >>>>>>> > > > >>>>>>> pt., 4 wrz 2020, 10:15 użytkownik Bertrand Delacretaz < > > > >>>>>>> bdelacre...@apache.org> napisał: > > > >>>>>>> > > > >>>>>>>> Hi, > > > >>>>>>>> > > > >>>>>>>> On Fri, Sep 4, 2020 at 9:22 AM Jarek Potiuk <ja...@potiuk.com > > > > > >>>> wrote: > > > >>>>>>>>> ...I think having ASF "apache/chart" for all "officially > > > >>>> released" helm > > > >>>>>>>>> charts is the best solution.... > > > >>>>>>>> > > > >>>>>>>> Sounds like a good idea but please use a more specific name, > > > >>>> "charts" > > > >>>>>>>> is very generic. > > > >>>>>>>> > > > >>>>>>>> If distributing those Helm charts does not requires more than > a > > > >>> web > > > >>>>>>>> server (with some metadata files I suppose) that can probably > > > >> go > > > >>>> in a > > > >>>>>>>> specific folder under > > > >>> https://dist.apache.org/repos/dist/release/ > > > >>>>>>>> > > > >>>>>>>> The next question is what kind of traffic is expected there, > > > >> and > > > >>>> can > > > >>>>>>>> the clients which will download those Helm charts handle > > > >>> redirects > > > >>>> to > > > >>>>>>>> distribution mirrors? If yes the mechanism of > > > >>>>>>>> http://www.apache.org/dev/release-download-pages.html comes > to > > > >>>> mind, > > > >>>>>>>> if you can piggyback on that mirroring infrastructure that's a > > > >>> big > > > >>>>>>>> plus IMO. > > > >>>>>>>> > > > >>>>>>>> I guess the way to move forward is for the people interested > in > > > >>>> this > > > >>>>>>>> to list the technical and licensing/legal requirements on a > > > >> wiki > > > >>>> page > > > >>>>>>>> or equivalent, to discuss with the ASF infrastructure team how > > > >> to > > > >>>> best > > > >>>>>>>> support them. > > > >>>>>>>> > > > >>>>>>>> -Bertrand > > > >>>>>>>> > > > >>>>>>>> > > > >>>> > > --------------------------------------------------------------------- > > > >>>>>>>> To unsubscribe, e-mail: dev-unsubscr...@community.apache.org > > > >>>>>>>> For additional commands, e-mail: > dev-h...@community.apache.org > > > >>>>>>>> > > > >>>>>>>> > > > >>>>>>> > > > >>>>>> > > > >>>>>> > > > >> > --------------------------------------------------------------------- > > > >>>>>> To unsubscribe, e-mail: dev-unsubscr...@community.apache.org > > > >>>>>> For additional commands, e-mail: dev-h...@community.apache.org > > > >>>>>> > > > >>>>>> > > > >>>>> > > > >>>>> > > --------------------------------------------------------------------- > > > >>>>> To unsubscribe, e-mail: dev-unsubscr...@community.apache.org > > > >>>>> For additional commands, e-mail: dev-h...@community.apache.org > > > >>>>> > > > >>>> > > > >>>> > > > >>>> -- > > > >>>> Matt Sicker <boa...@gmail.com> > > > >>>> > > > >>>> > > --------------------------------------------------------------------- > > > >>>> To unsubscribe, e-mail: dev-unsubscr...@community.apache.org > > > >>>> For additional commands, e-mail: dev-h...@community.apache.org > > > >>>> > > > >>>> > > > >>> > > > >> > > > >> > > > >> -- > > > >> +48 660 796 129 > > > >> > > > > > > > > > > > -- > +48 660 796 129 >
