Maybe I didn't put that clear enough, but my comment referred to the chart not the image. In fact I've been always running the stable/airflow chart with a custom built image, without any issues. Upgrading it to the official Airflow image seems like a simple PR, which I'm sure would get accepted.
For creating another helm chart I see no benefits, just downsides: - helm repo to maintain - helm repo must be added by the user - chart doesn't go through official helm quality checks - we don't get future helm CI enhancements - confusion among users Next, helm chart releases will have to be aligned not only with Airflow, but also changes to helm and Kubernetes APIs. In short, I believe it's better to upgrade/fix the existing chart than create a new one. On Sun, 13 Oct 2019, 09:39 Jarek Potiuk, <[email protected]> wrote: > Some more comments about Production-ready image and security/patches > aspect. > > I updated the motivation section in AIP-26 > < > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-26+-+Production-ready+Airflow+Docker+Image+and+helm+chart > > > to > explain why we are working on community-driven image. I also updated > description about releasing new Airflow image after Python security patches > have been released (hint - we automate it :). > > With the current CI image build process - whenever we rebuild image on > DockerHub we automatically take into account the latest security patches > released by Python. We can easily make it in the way that we rebuild and > re-publish even all the already released images to include those patches > (however we need to include some kind of semver versioning for that I > think). I make sure this is included while working on the POC PR for > PROD-ready image - https://github.com/apache/airflow/pull/6266 > > This solves problems of security patches of the base Python image, but one > thing that came to my mind is whether we should pin dependencies for those > released images (and master image)? This is also related to separate > pinning-dependencies > < > https://lists.apache.org/thread.html/c643d17833fd3bea534c8c6072cc91e7878348d79dcf756eb94d483f@%3Cdev.airflow.apache.org%3E > > > thread. > > My questio is - should we regularly re-release old images and re-pin the > dependencies with security patches? I guess so. > > I am already looking at some ways to automate (optional) pinning of > dependencies and possibly also automating of applying latest security > patches for dependencies. I think however that this should be a separate > AIP/proposal after we have the production image out and we can release the > production image for testing before we address it. > > I would love to hear your comments. > > J. > > > > On Sun, Oct 13, 2019 at 9:10 AM Jarek Potiuk <[email protected]> > wrote: > > > Sorry - I missed that comment - it was not that you got ignored, I just > > missed it in the flood of emails I had :(. sorry. > > > > The chart (and corresponding puckel image) is quite ok for the past but > if > > we want to move forward, we need to make sure that the image, charts etc. > > are driven and managed by the community following release schedule and > > processes of Apache Software Foundation. > > > > The current chart uses (often used) puckel image which was good for quite > > a while but it was not really part of the Apache official community > effort. > > For example one of the rules of releasing software is that any software > > formally released by the project must be voted by PMC ( > > https://www.apache.org/foundation/how-it-works.html#pmc-members) > > > > By bringing the official image to apache/airflow repository and making > > sure it is part of the release process of Airflow we can release new > images > > at the same time new versions of Airflow get released. Additionally we > can > > provide more maintainability - for example add some more detailed > > instructions and guidelines on how to run Airflow in the production > > environment. We can also make sure we have some optimisations in place > and > > support wider set of audience - hopefully we can get some feedback from > > people using the official Airflow image/chart and address it longer term. > > Once we incorporate it to our community process, it will be easier for > > everyone to contribute to it - in the same way they contribute to the > code > > of Airflow. > > > > I will update Motivation section in the AIP to include that. > > > > J. > > > > > > > > On Mon, Oct 7, 2019 at 8:06 PM Marcin Szymański <[email protected]> > wrote: > > > >> What are the issues with the existing stable/airflow chart in helm > public > >> repo? I've used it on a number of occasions and it's good enough to > deploy > >> Airflow with Celery executor and run Kubernetes pods from Airflow using > >> in_cluster mode. Not sure if it allows to deploy Kubernetes executor > >> though. > >> > >> On Mon, Oct 7, 2019 at 8:43 AM Jarek Potiuk <[email protected]> > >> wrote: > >> > >> > I just created "AIP-26 - Production-ready Airflow Docker Image and > helm > >> > chart > >> > < > >> > > >> > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-26+-+Production-ready+Airflow+Docker+Image+and+helm+chart > >> > >" > >> > proposal and would love community feedback and comments. > >> > > >> > I would love to know what is your expectation from the Docker image > and > >> how > >> > you are using production images currently. > >> > > >> > We were looking at the images available together with Daniel Imberman > >> > (including puckel and Astronomer image) and I started to work on a > draft > >> > POC > >> > <https://github.com/apache/airflow/pull/6266> to incorporate building > >> of > >> > the production-ready image into the CI framework we have. The current > >> POC > >> > uses the same Dockerfile we used for CI images but run with > >> > production-ready parameters. I made a number of refactorings and > >> > simplifications to the build logic and the scripts should be much more > >> sane > >> > now (and unit-testable soon). > >> > > >> > The images are based on Debian buster (Debian 10) LTS released few > >> months > >> > ago (so far we used Debian stretch for CI). It still needs some fixes > >> for > >> > failing tests (https://travis-ci.org/potiuk/airflow/builds/594281768) > >> but > >> > the image itself can be downloaded from my private repo: `docker pull > >> > potiuk/airflow:master-python3.5`) > >> > > >> > The Docker file is available here: > >> > https://github.com/PolideaInternal/airflow/blob/prod-image/Dockerfile > >> > > >> > Once we have good image we will work with Daniel on helm chart so that > >> > Airflow can be easily installed on Kubernetes. > >> > > >> > Any comments and feedback/discussion in the AIP-26 document are > welcome > >> ! > >> > > >> > J. > >> > > >> > -- > >> > > >> > Jarek Potiuk > >> > Polidea <https://www.polidea.com/> | Principal Software Engineer > >> > > >> > M: +48 660 796 129 <+48660796129> > >> > [image: Polidea] <https://www.polidea.com/> > >> > > >> > > > > > > -- > > > > Jarek Potiuk > > Polidea <https://www.polidea.com/> | Principal Software Engineer > > > > M: +48 660 796 129 <+48660796129> > > [image: Polidea] <https://www.polidea.com/> > > > > > > -- > > Jarek Potiuk > Polidea <https://www.polidea.com/> | Principal Software Engineer > > M: +48 660 796 129 <+48660796129> > [image: Polidea] <https://www.polidea.com/> >
