Issue created! https://github.com/helm/charts/issues/17933 . Thanks Jonathan for feedback and bringing this up!
On Mon, Oct 14, 2019 at 8:29 AM Jarek Potiuk <[email protected]> wrote: > Thanks! Good points Jonathan. I will loop in the OWNERS indeed. I am also > in contact with Daniel from Bitnami who manages the Bitnami Airflow image > and helm chart and hopefully we can work out an approach where we can work > together :). Helm Hub is also good. I updated the AIP to reflect those. > > On Mon, Oct 14, 2019 at 3:43 AM Jonathan Miles <[email protected]> wrote: > >> As a user of the official Helm Chart within The Trade Desk and >> contributor of a few things there, I'm happy that the intention is to >> reuse it. Like Marcin, I also got the opposite impression from reading >> the earlier thread messages, so thanks for clarifying. There are many >> valuable contributions to that chart from various people since it was >> merged back in December 2018. >> >> There are two "owners" for the chart registered at the moment: >> https://github.com/helm/charts/blob/master/stable/airflow/OWNERS. >> Suppose it will be some time until the new image is ready, but probably >> worth looping them in by creating a GitHub issue to register the intent >> and discuss? Gsemet was the most responsive when I made my contributions. >> >> Now that there's also Helm Hub - https://hub.helm.sh/charts?q=airflow - >> to help deal some of the scale challenges of managing so many charts in >> a monorepo, it might also be worth considering moving the chart to its >> own repo. It'd be easier to couple the chart to official Airflow >> versions and even have multiple branches/tags in parallel. >> >> Jon >> >> On 13/10/2019 10:09, Jarek Potiuk wrote: >> > It's likely we are both talking about the same :). >> > >> > I was never proposing to create a new helm chart to Apache Airflow >> repo. I >> > just proposed that the official docker image is made part of Airflow and >> > change the official helm chart to use it. >> > >> > Official Helm charts are distributed via the helm github repo: >> > https://github.com/helm/charts/tree/master/stable/airflow and it >> should >> > remain like that, no doubt about it. Maybe we will have to make a few >> > adjustments to the chart itself to make it work with the new Airflow >> image, >> > but it likely should remain unchanged. There are multiple references to >> > puckel image in the chart docs (how to fork etc.) and they should be >> > updated to the official image. I believe Daniel Imberman wanted to work >> on >> > the helm chart once the docker image is released (I am happy to help >> with >> > that as well). >> > >> > I will make sure to update that in the AIP so that it is clear we are >> not >> > going to have another copy of the chart. >> > >> > J. >> > >> > >> > On Sun, Oct 13, 2019 at 10:57 AM Marcin Szymański <[email protected]> >> wrote: >> > >> >> Maybe I didn't put that clear enough, but my comment referred to the >> chart >> >> not the image. In fact I've been always running the stable/airflow >> chart >> >> with a custom built image, without any issues. Upgrading it to the >> official >> >> Airflow image seems like a simple PR, which I'm sure would get >> accepted. >> >> >> >> For creating another helm chart I see no benefits, just downsides: >> >> - helm repo to maintain >> >> - helm repo must be added by the user >> >> - chart doesn't go through official helm quality checks >> >> - we don't get future helm CI enhancements >> >> - confusion among users >> >> >> >> Next, helm chart releases will have to be aligned not only with >> Airflow, >> >> but also changes to helm and Kubernetes APIs. >> >> >> >> In short, I believe it's better to upgrade/fix the existing chart than >> >> create a new one. >> >> >> >> On Sun, 13 Oct 2019, 09:39 Jarek Potiuk, <[email protected]> >> wrote: >> >> >> >>> Some more comments about Production-ready image and security/patches >> >>> aspect. >> >>> >> >>> I updated the motivation section in AIP-26 >> >>> < >> >>> >> >> >> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-26+-+Production-ready+Airflow+Docker+Image+and+helm+chart >> >>> to >> >>> explain why we are working on community-driven image. I also updated >> >>> description about releasing new Airflow image after Python security >> >> patches >> >>> have been released (hint - we automate it :). >> >>> >> >>> With the current CI image build process - whenever we rebuild image on >> >>> DockerHub we automatically take into account the latest security >> patches >> >>> released by Python. We can easily make it in the way that we rebuild >> and >> >>> re-publish even all the already released images to include those >> patches >> >>> (however we need to include some kind of semver versioning for that I >> >>> think). I make sure this is included while working on the POC PR for >> >>> PROD-ready image - https://github.com/apache/airflow/pull/6266 >> >>> >> >>> This solves problems of security patches of the base Python image, but >> >> one >> >>> thing that came to my mind is whether we should pin dependencies for >> >> those >> >>> released images (and master image)? This is also related to separate >> >>> pinning-dependencies >> >>> < >> >>> >> >> >> https://lists.apache.org/thread.html/c643d17833fd3bea534c8c6072cc91e7878348d79dcf756eb94d483f@%3Cdev.airflow.apache.org%3E >> >>> thread. >> >>> >> >>> My questio is - should we regularly re-release old images and re-pin >> the >> >>> dependencies with security patches? I guess so. >> >>> >> >>> I am already looking at some ways to automate (optional) pinning of >> >>> dependencies and possibly also automating of applying latest security >> >>> patches for dependencies. I think however that this should be a >> separate >> >>> AIP/proposal after we have the production image out and we can release >> >> the >> >>> production image for testing before we address it. >> >>> >> >>> I would love to hear your comments. >> >>> >> >>> J. >> >>> >> >>> >> >>> >> >>> On Sun, Oct 13, 2019 at 9:10 AM Jarek Potiuk < >> [email protected]> >> >>> wrote: >> >>> >> >>>> Sorry - I missed that comment - it was not that you got ignored, I >> just >> >>>> missed it in the flood of emails I had :(. sorry. >> >>>> >> >>>> The chart (and corresponding puckel image) is quite ok for the past >> but >> >>> if >> >>>> we want to move forward, we need to make sure that the image, charts >> >> etc. >> >>>> are driven and managed by the community following release schedule >> and >> >>>> processes of Apache Software Foundation. >> >>>> >> >>>> The current chart uses (often used) puckel image which was good for >> >> quite >> >>>> a while but it was not really part of the Apache official community >> >>> effort. >> >>>> For example one of the rules of releasing software is that any >> software >> >>>> formally released by the project must be voted by PMC ( >> >>>> https://www.apache.org/foundation/how-it-works.html#pmc-members) >> >>>> >> >>>> By bringing the official image to apache/airflow repository and >> making >> >>>> sure it is part of the release process of Airflow we can release new >> >>> images >> >>>> at the same time new versions of Airflow get released. Additionally >> we >> >>> can >> >>>> provide more maintainability - for example add some more detailed >> >>>> instructions and guidelines on how to run Airflow in the production >> >>>> environment. We can also make sure we have some optimisations in >> place >> >>> and >> >>>> support wider set of audience - hopefully we can get some feedback >> from >> >>>> people using the official Airflow image/chart and address it longer >> >> term. >> >>>> Once we incorporate it to our community process, it will be easier >> for >> >>>> everyone to contribute to it - in the same way they contribute to the >> >>> code >> >>>> of Airflow. >> >>>> >> >>>> I will update Motivation section in the AIP to include that. >> >>>> >> >>>> J. >> >>>> >> >>>> >> >>>> >> >>>> On Mon, Oct 7, 2019 at 8:06 PM Marcin Szymański <[email protected]> >> >>> wrote: >> >>>>> What are the issues with the existing stable/airflow chart in helm >> >>> public >> >>>>> repo? I've used it on a number of occasions and it's good enough to >> >>> deploy >> >>>>> Airflow with Celery executor and run Kubernetes pods from Airflow >> >> using >> >>>>> in_cluster mode. Not sure if it allows to deploy Kubernetes executor >> >>>>> though. >> >>>>> >> >>>>> On Mon, Oct 7, 2019 at 8:43 AM Jarek Potiuk < >> [email protected] >> >>>>> wrote: >> >>>>> >> >>>>>> I just created "AIP-26 - Production-ready Airflow Docker Image and >> >>> helm >> >>>>>> chart >> >>>>>> < >> >>>>>> >> >> >> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-26+-+Production-ready+Airflow+Docker+Image+and+helm+chart >> >>>>>>> " >> >>>>>> proposal and would love community feedback and comments. >> >>>>>> >> >>>>>> I would love to know what is your expectation from the Docker image >> >>> and >> >>>>> how >> >>>>>> you are using production images currently. >> >>>>>> >> >>>>>> We were looking at the images available together with Daniel >> >> Imberman >> >>>>>> (including puckel and Astronomer image) and I started to work on a >> >>> draft >> >>>>>> POC >> >>>>>> <https://github.com/apache/airflow/pull/6266> to incorporate >> >> building >> >>>>> of >> >>>>>> the production-ready image into the CI framework we have. The >> >> current >> >>>>> POC >> >>>>>> uses the same Dockerfile we used for CI images but run with >> >>>>>> production-ready parameters. I made a number of refactorings and >> >>>>>> simplifications to the build logic and the scripts should be much >> >> more >> >>>>> sane >> >>>>>> now (and unit-testable soon). >> >>>>>> >> >>>>>> The images are based on Debian buster (Debian 10) LTS released few >> >>>>> months >> >>>>>> ago (so far we used Debian stretch for CI). It still needs some >> >> fixes >> >>>>> for >> >>>>>> failing tests ( >> >> https://travis-ci.org/potiuk/airflow/builds/594281768) >> >>>>> but >> >>>>>> the image itself can be downloaded from my private repo: `docker >> >> pull >> >>>>>> potiuk/airflow:master-python3.5`) >> >>>>>> >> >>>>>> The Docker file is available here: >> >>>>>> >> >> https://github.com/PolideaInternal/airflow/blob/prod-image/Dockerfile >> >>>>>> Once we have good image we will work with Daniel on helm chart so >> >> that >> >>>>>> Airflow can be easily installed on Kubernetes. >> >>>>>> >> >>>>>> Any comments and feedback/discussion in the AIP-26 document are >> >>> welcome >> >>>>> ! >> >>>>>> J. >> >>>>>> >> >>>>>> -- >> >>>>>> >> >>>>>> Jarek Potiuk >> >>>>>> Polidea <https://www.polidea.com/> | Principal Software Engineer >> >>>>>> >> >>>>>> M: +48 660 796 129 <+48660796129> >> >>>>>> [image: Polidea] <https://www.polidea.com/> >> >>>>>> >> >>>> >> >>>> -- >> >>>> >> >>>> Jarek Potiuk >> >>>> Polidea <https://www.polidea.com/> | Principal Software Engineer >> >>>> >> >>>> M: +48 660 796 129 <+48660796129> >> >>>> [image: Polidea] <https://www.polidea.com/> >> >>>> >> >>>> >> >>> -- >> >>> >> >>> Jarek Potiuk >> >>> Polidea <https://www.polidea.com/> | Principal Software Engineer >> >>> >> >>> M: +48 660 796 129 <+48660796129> >> >>> [image: Polidea] <https://www.polidea.com/> >> >>> >> > >> > > > -- > > Jarek Potiuk > Polidea <https://www.polidea.com/> | Principal Software Engineer > > M: +48 660 796 129 <+48660796129> > [image: Polidea] <https://www.polidea.com/> > > -- Jarek Potiuk Polidea <https://www.polidea.com/> | Principal Software Engineer M: +48 660 796 129 <+48660796129> [image: Polidea] <https://www.polidea.com/>
