I've been thinking a lot and discussing the issues mentioned here (as in Apache Airflow we had this very case to solve. I would like to extend what Kamil wrote - and comment on what Apache Airflow uses and some of the recent discussions we had. I think we had very recently a lot of discussions about the Helm chart, licencing, releasing mechanisms so that might be a useful experience for everyone.
TL;DR; I think it is a GREAT idea to have a single place from where all ASF charts will be hosted. That has a number of advantages and solves a number of pain points we experienced so far with the distribution mechanism of the Helm Charts and I think it's a great opportunity to plug a small "ambiguity" in the ASF release policy - coming from the way how Helm and Docker Images in general are a bit different than the "existing release mechanism". This is not a big ambiguity IMHO and the current rules of the release policy can be interpreted and applied to Helm Chart policies, but it would be simply great to make clear rules about both - images and charts as they will likely become a very important distribution and release mechanism in the near future (if not already did). Some ideas/proposals resulting from the discussions I raised: 1) I think having ASF "apache/chart" for all "officially released" helm charts is the best solution. This should not be a place where the helm charts are developed - but you should be able to get a released and tagged versions of those charts. Each project should have their own way of developing the chart (might be in mono-repo with other parts of the system or separate repo). For example in Airflow we are developing chart together with main Airflow code and we are using it during the CI to run tests - effectively testing "official image", "airflow code" and "airlfow chart" from the latest master together - so we chose to keep monorepo with all those. But for other projects, they might have "apache/project-chart" repo with only the chart. This should be project-based decision. 2) There are already tools (such as https://github.com/google/copybara) that allow to copy selectively parts of the repositories. And with Github Actions, we could very easily write an automation where each project could configure where their chart repository is, which folder, branch and tags are used and the common "ASF Chart" repo could easily pull-in such charts. This way releases marked as tags in the original projects would automatically land in the "ASF Chart" repo - also some validation might be put in place to make sure the pulled chart is "good to go" and some notifications might be in place if they are not. I am super-happy to help with such automation - I've done a lot of Github Actions work recently for Apache Airflow and helped Apache Beam as well - including developing a very useful https://github.com/marketplace/actions/cancel-workflow-runs action that cancels duplicate Github Actions Workflow and helps us to optimise our Job usage and CI feedback time by 30%. 3) Regarding the licencing, policies - I started recently a discussion about this very subject https://lists.apache.org/thread.html/rf2af2a95e7687fe94ede23fe9df388f784c8231a5968b109f677cbe8%40%3Cbuilds.apache.org%3E and there was no "definittive" or "clear" answers. There is no direct mentioning of Helm or Docker release mechanisms in the http://www.apache.org/legal/release-policy.html but IMHO we can interpret some of the statements there. I will put relevant parts from the policy with my interpretation: a) Docker Images are - rather clearly - binary convenience packages and we are free to provide them to the users as such - as long we also give users a way to recreate them as long as they have necessary platform and tools. So all the sources required to rebuild those images MUST be released as an "official source package" - following all the rules of ASF (voting, signing etc.). According to that - I think there is no need to vote on releasing the Docker Images as long as they are build the same sources that have been officially voted and released. > As a convenience to users that might not have the appropriate tools to build a compiled version of the source, binary/bytecode packages MAY be distributed alongside official Apache releases. In all such cases, the binary/bytecode package MUST have the same version number as the source release and MUST only add binary/bytecode files that are the result of compiling that version of the source code release and its dependencies. b) Helm Chart sources - they are sources not binary releases. So sources of the Helm Chart should be officially voted, signed and released. However the official release process should be the regular "release process" and can be made part of the source package in "downloads.apache.org". In case of Airflow's monorepo for example - we will just include helm chart sources next time when we release Airflow 1.10.13 and this will also be the "official" Helm Chart release. We are tagging the source code with "1.10.13" tag. And this tag might then be used to "release" the helm chart in the "ASF chart" directory. So the "ASF Chart" release will not require separate voting as this will - again - be a "convenience" release for our users. As long as "official" releases are put there using the same sources that were officially released in "SVN" - we are perfectly fine with the release policy of ASF IMHO c) Docker images USED by the helm chart: There is one question though which I raised in the discussion above. Helm Charts often use external images (not only images of the ASF project). And I think - this is something that I think is not entirely clear - that our users MUST be able to rebuild those images using only the sources released by the project officially + official base images that the Docker Images are built from. There are many cases where the sources for Docker Images are maintained (or not) by random people - because it is so easy to point to an existing DockerHub image somewhere without worrying how the image is built. But I personally think this is against this very rule written in ASF release policy: "Every ASF release must contain a source package, which must be sufficient for a user to build and test the release provided they have access to the appropriate platform and tool" Therefore in Airflow I proposed and we merged-in the change ( https://github.com/apache/airflow/pull/9650) where sources for all the images that we are using are placed under the "Apache Airflow" control and with the next release of Airflow we are going to include all the sources in the "official source package" so that we can release the Helm Chart "officially" as well. So my proposal is that this "compliance" with "use only officially released sources" and "can be rebuild by the users" becomes a requirement for all ASF sources to be hosted in the "ASF Helm Chart". I think that might be great way to define a "policy" about images and charts that all ASF projects might follow. I am super-happy to help in making this happen - I can work on all automation and release mechanisms for the "ASF Helm Chart" if we decide that this is the way to go. Happy to put a lot of effort in it. J, On Fri, Sep 4, 2020 at 5:48 AM Matt Sicker <boa...@gmail.com> wrote: > If Helm charts have similar licensing complexities as Docker images, then > it probably makes sense to migrate charts to individual projects rather > than making a larger chart repository here at Apache. If they don’t have > similar licensing problems, then it could be aggregated like the Docker > org, though that’s an orthogonal concern at this point based on how Helm > charts have migrated from the centralized distribution model. Helm makes it > fairly easy to add chart repositories (at least in v3) so that individual > projects can own their own charts. > > On Thu, Sep 3, 2020 at 22:32 Dave Fisher <wave4d...@comcast.net> wrote: > > > The ASF and the Apache Way is all about the community around an open > > source software project. If there is no community around this deprecated > > package then you will need to contact each Apache community that might be > > interested and ask them directly. > > > > > > > > If you have a group of developers who want to preserve this project then > > incubating an Apache project could be an approach. > > > > > > > > Regards, > > > > Dave > > > > > > > > Sent from my iPhone > > > > > > > > > On Sep 3, 2020, at 7:00 PM, Ween Jiann <wjlee.2...@smu.edu.sg> wrote: > > > > > > > > > > Hi Dave, > > > > > > > > > > Just to clarify, my suggestion is to move only charts related to ASF > > projects to the apache git repo. > > > > > These charts have been deprecated as part of the repo deprecation, they > > have no choice but to move somewhere or be archived. > > > > > > > > > > If this gains some traction, I'll create an issue and contact the > > maintainers of the charts. > > > > > > > > > >> On 2020/09/03 21:01:12, Dave Fisher <wave4d...@comcast.net> wrote: > > > > >> Hi - > > > > >> > > > > >> Does the Helm Chart community wish to explore becoming an Apache > > Project Community? > > > > >> > > > > >> Regards, > > > > >> Dave > > > > >> > > > > >> Sent from my iPhone > > > > >> > > > > >>>> On Sep 3, 2020, at 12:33 PM, Ween Jiann <wjlee.2...@smu.edu.sg> > > wrote: > > > > >>> > > > > >>> Hi all, > > > > >>> > > > > >>> The helm team has deprecated the charts repository and will be > > archiving it in Nov. Here’s the link to the notice > > > https://github.com/helm/charts/blob/master/README.md#deprecation-timeline > > > > >>> > > > > >>> It looks like that are quite a few charts for ASF projects such as > > Airflow, Hadoop, Spark, Solr (not exhaustive) that have yet to be adopted > > by anyone. It would make sense to re-home them into a single git in for > > example apache/charts. This would facilitate easier distribution of > various > > ASF projects via a single helm repo, and setting up tests and CI > > integration needs only to be done once. > > > > >>> > > > > >>> Cheers, > > > > >>> WJ > > > > >>> > > > > >>> --------------------------------------------------------------------- > > > > >>> To unsubscribe, e-mail: dev-unsubscr...@community.apache.org > > > > >>> For additional commands, e-mail: dev-h...@community.apache.org > > > > >>> > > > > >> > > > > >> > > > > >> --------------------------------------------------------------------- > > > > >> To unsubscribe, e-mail: dev-unsubscr...@community.apache.org > > > > >> For additional commands, e-mail: dev-h...@community.apache.org > > > > >> > > > > >> > > > > > > > > > > --------------------------------------------------------------------- > > > > > To unsubscribe, e-mail: dev-unsubscr...@community.apache.org > > > > > For additional commands, e-mail: dev-h...@community.apache.org > > > > > > > > > > > > > > > > > --------------------------------------------------------------------- > > > > To unsubscribe, e-mail: dev-unsubscr...@community.apache.org > > > > For additional commands, e-mail: dev-h...@community.apache.org > > > > > > > > -- > Matt Sicker <boa...@gmail.com> > -- +48 660 796 129
