Hello everyone, I think we had some useful feedback and good points raised/reflected in the document. I think there is one remaining question about pinning dependencies and upgrading PIP dependencies with security fixes, but I think we can make it separate AIP. Are there any other concerns, or should I start a vote on it ?
J. On Mon, Oct 14, 2019 at 8:42 AM Jarek Potiuk <[email protected]> wrote: > Issue created! https://github.com/helm/charts/issues/17933 . Thanks > Jonathan for feedback and bringing this up! > > On Mon, Oct 14, 2019 at 8:29 AM Jarek Potiuk <[email protected]> > wrote: > >> Thanks! Good points Jonathan. I will loop in the OWNERS indeed. I am also >> in contact with Daniel from Bitnami who manages the Bitnami Airflow image >> and helm chart and hopefully we can work out an approach where we can work >> together :). Helm Hub is also good. I updated the AIP to reflect those. >> >> On Mon, Oct 14, 2019 at 3:43 AM Jonathan Miles <[email protected]> wrote: >> >>> As a user of the official Helm Chart within The Trade Desk and >>> contributor of a few things there, I'm happy that the intention is to >>> reuse it. Like Marcin, I also got the opposite impression from reading >>> the earlier thread messages, so thanks for clarifying. There are many >>> valuable contributions to that chart from various people since it was >>> merged back in December 2018. >>> >>> There are two "owners" for the chart registered at the moment: >>> https://github.com/helm/charts/blob/master/stable/airflow/OWNERS. >>> Suppose it will be some time until the new image is ready, but probably >>> worth looping them in by creating a GitHub issue to register the intent >>> and discuss? Gsemet was the most responsive when I made my contributions. >>> >>> Now that there's also Helm Hub - https://hub.helm.sh/charts?q=airflow - >>> to help deal some of the scale challenges of managing so many charts in >>> a monorepo, it might also be worth considering moving the chart to its >>> own repo. It'd be easier to couple the chart to official Airflow >>> versions and even have multiple branches/tags in parallel. >>> >>> Jon >>> >>> On 13/10/2019 10:09, Jarek Potiuk wrote: >>> > It's likely we are both talking about the same :). >>> > >>> > I was never proposing to create a new helm chart to Apache Airflow >>> repo. I >>> > just proposed that the official docker image is made part of Airflow >>> and >>> > change the official helm chart to use it. >>> > >>> > Official Helm charts are distributed via the helm github repo: >>> > https://github.com/helm/charts/tree/master/stable/airflow and it >>> should >>> > remain like that, no doubt about it. Maybe we will have to make a few >>> > adjustments to the chart itself to make it work with the new Airflow >>> image, >>> > but it likely should remain unchanged. There are multiple references to >>> > puckel image in the chart docs (how to fork etc.) and they should be >>> > updated to the official image. I believe Daniel Imberman wanted to >>> work on >>> > the helm chart once the docker image is released (I am happy to help >>> with >>> > that as well). >>> > >>> > I will make sure to update that in the AIP so that it is clear we are >>> not >>> > going to have another copy of the chart. >>> > >>> > J. >>> > >>> > >>> > On Sun, Oct 13, 2019 at 10:57 AM Marcin Szymański <[email protected]> >>> wrote: >>> > >>> >> Maybe I didn't put that clear enough, but my comment referred to the >>> chart >>> >> not the image. In fact I've been always running the stable/airflow >>> chart >>> >> with a custom built image, without any issues. Upgrading it to the >>> official >>> >> Airflow image seems like a simple PR, which I'm sure would get >>> accepted. >>> >> >>> >> For creating another helm chart I see no benefits, just downsides: >>> >> - helm repo to maintain >>> >> - helm repo must be added by the user >>> >> - chart doesn't go through official helm quality checks >>> >> - we don't get future helm CI enhancements >>> >> - confusion among users >>> >> >>> >> Next, helm chart releases will have to be aligned not only with >>> Airflow, >>> >> but also changes to helm and Kubernetes APIs. >>> >> >>> >> In short, I believe it's better to upgrade/fix the existing chart than >>> >> create a new one. >>> >> >>> >> On Sun, 13 Oct 2019, 09:39 Jarek Potiuk, <[email protected]> >>> wrote: >>> >> >>> >>> Some more comments about Production-ready image and security/patches >>> >>> aspect. >>> >>> >>> >>> I updated the motivation section in AIP-26 >>> >>> < >>> >>> >>> >> >>> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-26+-+Production-ready+Airflow+Docker+Image+and+helm+chart >>> >>> to >>> >>> explain why we are working on community-driven image. I also updated >>> >>> description about releasing new Airflow image after Python security >>> >> patches >>> >>> have been released (hint - we automate it :). >>> >>> >>> >>> With the current CI image build process - whenever we rebuild image >>> on >>> >>> DockerHub we automatically take into account the latest security >>> patches >>> >>> released by Python. We can easily make it in the way that we rebuild >>> and >>> >>> re-publish even all the already released images to include those >>> patches >>> >>> (however we need to include some kind of semver versioning for that I >>> >>> think). I make sure this is included while working on the POC PR for >>> >>> PROD-ready image - https://github.com/apache/airflow/pull/6266 >>> >>> >>> >>> This solves problems of security patches of the base Python image, >>> but >>> >> one >>> >>> thing that came to my mind is whether we should pin dependencies for >>> >> those >>> >>> released images (and master image)? This is also related to separate >>> >>> pinning-dependencies >>> >>> < >>> >>> >>> >> >>> https://lists.apache.org/thread.html/c643d17833fd3bea534c8c6072cc91e7878348d79dcf756eb94d483f@%3Cdev.airflow.apache.org%3E >>> >>> thread. >>> >>> >>> >>> My questio is - should we regularly re-release old images and re-pin >>> the >>> >>> dependencies with security patches? I guess so. >>> >>> >>> >>> I am already looking at some ways to automate (optional) pinning of >>> >>> dependencies and possibly also automating of applying latest security >>> >>> patches for dependencies. I think however that this should be a >>> separate >>> >>> AIP/proposal after we have the production image out and we can >>> release >>> >> the >>> >>> production image for testing before we address it. >>> >>> >>> >>> I would love to hear your comments. >>> >>> >>> >>> J. >>> >>> >>> >>> >>> >>> >>> >>> On Sun, Oct 13, 2019 at 9:10 AM Jarek Potiuk < >>> [email protected]> >>> >>> wrote: >>> >>> >>> >>>> Sorry - I missed that comment - it was not that you got ignored, I >>> just >>> >>>> missed it in the flood of emails I had :(. sorry. >>> >>>> >>> >>>> The chart (and corresponding puckel image) is quite ok for the past >>> but >>> >>> if >>> >>>> we want to move forward, we need to make sure that the image, charts >>> >> etc. >>> >>>> are driven and managed by the community following release schedule >>> and >>> >>>> processes of Apache Software Foundation. >>> >>>> >>> >>>> The current chart uses (often used) puckel image which was good for >>> >> quite >>> >>>> a while but it was not really part of the Apache official community >>> >>> effort. >>> >>>> For example one of the rules of releasing software is that any >>> software >>> >>>> formally released by the project must be voted by PMC ( >>> >>>> https://www.apache.org/foundation/how-it-works.html#pmc-members) >>> >>>> >>> >>>> By bringing the official image to apache/airflow repository and >>> making >>> >>>> sure it is part of the release process of Airflow we can release new >>> >>> images >>> >>>> at the same time new versions of Airflow get released. Additionally >>> we >>> >>> can >>> >>>> provide more maintainability - for example add some more detailed >>> >>>> instructions and guidelines on how to run Airflow in the production >>> >>>> environment. We can also make sure we have some optimisations in >>> place >>> >>> and >>> >>>> support wider set of audience - hopefully we can get some feedback >>> from >>> >>>> people using the official Airflow image/chart and address it longer >>> >> term. >>> >>>> Once we incorporate it to our community process, it will be easier >>> for >>> >>>> everyone to contribute to it - in the same way they contribute to >>> the >>> >>> code >>> >>>> of Airflow. >>> >>>> >>> >>>> I will update Motivation section in the AIP to include that. >>> >>>> >>> >>>> J. >>> >>>> >>> >>>> >>> >>>> >>> >>>> On Mon, Oct 7, 2019 at 8:06 PM Marcin Szymański <[email protected]> >>> >>> wrote: >>> >>>>> What are the issues with the existing stable/airflow chart in helm >>> >>> public >>> >>>>> repo? I've used it on a number of occasions and it's good enough to >>> >>> deploy >>> >>>>> Airflow with Celery executor and run Kubernetes pods from Airflow >>> >> using >>> >>>>> in_cluster mode. Not sure if it allows to deploy Kubernetes >>> executor >>> >>>>> though. >>> >>>>> >>> >>>>> On Mon, Oct 7, 2019 at 8:43 AM Jarek Potiuk < >>> [email protected] >>> >>>>> wrote: >>> >>>>> >>> >>>>>> I just created "AIP-26 - Production-ready Airflow Docker Image and >>> >>> helm >>> >>>>>> chart >>> >>>>>> < >>> >>>>>> >>> >> >>> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-26+-+Production-ready+Airflow+Docker+Image+and+helm+chart >>> >>>>>>> " >>> >>>>>> proposal and would love community feedback and comments. >>> >>>>>> >>> >>>>>> I would love to know what is your expectation from the Docker >>> image >>> >>> and >>> >>>>> how >>> >>>>>> you are using production images currently. >>> >>>>>> >>> >>>>>> We were looking at the images available together with Daniel >>> >> Imberman >>> >>>>>> (including puckel and Astronomer image) and I started to work on a >>> >>> draft >>> >>>>>> POC >>> >>>>>> <https://github.com/apache/airflow/pull/6266> to incorporate >>> >> building >>> >>>>> of >>> >>>>>> the production-ready image into the CI framework we have. The >>> >> current >>> >>>>> POC >>> >>>>>> uses the same Dockerfile we used for CI images but run with >>> >>>>>> production-ready parameters. I made a number of refactorings and >>> >>>>>> simplifications to the build logic and the scripts should be much >>> >> more >>> >>>>> sane >>> >>>>>> now (and unit-testable soon). >>> >>>>>> >>> >>>>>> The images are based on Debian buster (Debian 10) LTS released few >>> >>>>> months >>> >>>>>> ago (so far we used Debian stretch for CI). It still needs some >>> >> fixes >>> >>>>> for >>> >>>>>> failing tests ( >>> >> https://travis-ci.org/potiuk/airflow/builds/594281768) >>> >>>>> but >>> >>>>>> the image itself can be downloaded from my private repo: `docker >>> >> pull >>> >>>>>> potiuk/airflow:master-python3.5`) >>> >>>>>> >>> >>>>>> The Docker file is available here: >>> >>>>>> >>> >> https://github.com/PolideaInternal/airflow/blob/prod-image/Dockerfile >>> >>>>>> Once we have good image we will work with Daniel on helm chart so >>> >> that >>> >>>>>> Airflow can be easily installed on Kubernetes. >>> >>>>>> >>> >>>>>> Any comments and feedback/discussion in the AIP-26 document are >>> >>> welcome >>> >>>>> ! >>> >>>>>> J. >>> >>>>>> >>> >>>>>> -- >>> >>>>>> >>> >>>>>> Jarek Potiuk >>> >>>>>> Polidea <https://www.polidea.com/> | Principal Software Engineer >>> >>>>>> >>> >>>>>> M: +48 660 796 129 <+48660796129> >>> >>>>>> [image: Polidea] <https://www.polidea.com/> >>> >>>>>> >>> >>>> >>> >>>> -- >>> >>>> >>> >>>> Jarek Potiuk >>> >>>> Polidea <https://www.polidea.com/> | Principal Software Engineer >>> >>>> >>> >>>> M: +48 660 796 129 <+48660796129> >>> >>>> [image: Polidea] <https://www.polidea.com/> >>> >>>> >>> >>>> >>> >>> -- >>> >>> >>> >>> Jarek Potiuk >>> >>> Polidea <https://www.polidea.com/> | Principal Software Engineer >>> >>> >>> >>> M: +48 660 796 129 <+48660796129> >>> >>> [image: Polidea] <https://www.polidea.com/> >>> >>> >>> > >>> >> >> >> -- >> >> Jarek Potiuk >> Polidea <https://www.polidea.com/> | Principal Software Engineer >> >> M: +48 660 796 129 <+48660796129> >> [image: Polidea] <https://www.polidea.com/> >> >> > > -- > > Jarek Potiuk > Polidea <https://www.polidea.com/> | Principal Software Engineer > > M: +48 660 796 129 <+48660796129> > [image: Polidea] <https://www.polidea.com/> > > -- Jarek Potiuk Polidea <https://www.polidea.com/> | Principal Software Engineer M: +48 660 796 129 <+48660796129> [image: Polidea] <https://www.polidea.com/>
