Sounds great for me. Thanks On Sun, 13 Oct 2019, 11:16 Jarek Potiuk, <[email protected]> wrote:
> It's likely we are both talking about the same :). > > I was never proposing to create a new helm chart to Apache Airflow repo. I > just proposed that the official docker image is made part of Airflow and > change the official helm chart to use it. > > Official Helm charts are distributed via the helm github repo: > https://github.com/helm/charts/tree/master/stable/airflow and it should > remain like that, no doubt about it. Maybe we will have to make a few > adjustments to the chart itself to make it work with the new Airflow image, > but it likely should remain unchanged. There are multiple references to > puckel image in the chart docs (how to fork etc.) and they should be > updated to the official image. I believe Daniel Imberman wanted to work on > the helm chart once the docker image is released (I am happy to help with > that as well). > > I will make sure to update that in the AIP so that it is clear we are not > going to have another copy of the chart. > > J. > > > On Sun, Oct 13, 2019 at 10:57 AM Marcin Szymański <[email protected]> > wrote: > > > Maybe I didn't put that clear enough, but my comment referred to the > chart > > not the image. In fact I've been always running the stable/airflow chart > > with a custom built image, without any issues. Upgrading it to the > official > > Airflow image seems like a simple PR, which I'm sure would get accepted. > > > > For creating another helm chart I see no benefits, just downsides: > > - helm repo to maintain > > - helm repo must be added by the user > > - chart doesn't go through official helm quality checks > > - we don't get future helm CI enhancements > > - confusion among users > > > > Next, helm chart releases will have to be aligned not only with Airflow, > > but also changes to helm and Kubernetes APIs. > > > > In short, I believe it's better to upgrade/fix the existing chart than > > create a new one. > > > > On Sun, 13 Oct 2019, 09:39 Jarek Potiuk, <[email protected]> > wrote: > > > > > Some more comments about Production-ready image and security/patches > > > aspect. > > > > > > I updated the motivation section in AIP-26 > > > < > > > > > > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-26+-+Production-ready+Airflow+Docker+Image+and+helm+chart > > > > > > > to > > > explain why we are working on community-driven image. I also updated > > > description about releasing new Airflow image after Python security > > patches > > > have been released (hint - we automate it :). > > > > > > With the current CI image build process - whenever we rebuild image on > > > DockerHub we automatically take into account the latest security > patches > > > released by Python. We can easily make it in the way that we rebuild > and > > > re-publish even all the already released images to include those > patches > > > (however we need to include some kind of semver versioning for that I > > > think). I make sure this is included while working on the POC PR for > > > PROD-ready image - https://github.com/apache/airflow/pull/6266 > > > > > > This solves problems of security patches of the base Python image, but > > one > > > thing that came to my mind is whether we should pin dependencies for > > those > > > released images (and master image)? This is also related to separate > > > pinning-dependencies > > > < > > > > > > https://lists.apache.org/thread.html/c643d17833fd3bea534c8c6072cc91e7878348d79dcf756eb94d483f@%3Cdev.airflow.apache.org%3E > > > > > > > thread. > > > > > > My questio is - should we regularly re-release old images and re-pin > the > > > dependencies with security patches? I guess so. > > > > > > I am already looking at some ways to automate (optional) pinning of > > > dependencies and possibly also automating of applying latest security > > > patches for dependencies. I think however that this should be a > separate > > > AIP/proposal after we have the production image out and we can release > > the > > > production image for testing before we address it. > > > > > > I would love to hear your comments. > > > > > > J. > > > > > > > > > > > > On Sun, Oct 13, 2019 at 9:10 AM Jarek Potiuk <[email protected] > > > > > wrote: > > > > > > > Sorry - I missed that comment - it was not that you got ignored, I > just > > > > missed it in the flood of emails I had :(. sorry. > > > > > > > > The chart (and corresponding puckel image) is quite ok for the past > but > > > if > > > > we want to move forward, we need to make sure that the image, charts > > etc. > > > > are driven and managed by the community following release schedule > and > > > > processes of Apache Software Foundation. > > > > > > > > The current chart uses (often used) puckel image which was good for > > quite > > > > a while but it was not really part of the Apache official community > > > effort. > > > > For example one of the rules of releasing software is that any > software > > > > formally released by the project must be voted by PMC ( > > > > https://www.apache.org/foundation/how-it-works.html#pmc-members) > > > > > > > > By bringing the official image to apache/airflow repository and > making > > > > sure it is part of the release process of Airflow we can release new > > > images > > > > at the same time new versions of Airflow get released. Additionally > we > > > can > > > > provide more maintainability - for example add some more detailed > > > > instructions and guidelines on how to run Airflow in the production > > > > environment. We can also make sure we have some optimisations in > place > > > and > > > > support wider set of audience - hopefully we can get some feedback > from > > > > people using the official Airflow image/chart and address it longer > > term. > > > > Once we incorporate it to our community process, it will be easier > for > > > > everyone to contribute to it - in the same way they contribute to the > > > code > > > > of Airflow. > > > > > > > > I will update Motivation section in the AIP to include that. > > > > > > > > J. > > > > > > > > > > > > > > > > On Mon, Oct 7, 2019 at 8:06 PM Marcin Szymański <[email protected]> > > > wrote: > > > > > > > >> What are the issues with the existing stable/airflow chart in helm > > > public > > > >> repo? I've used it on a number of occasions and it's good enough to > > > deploy > > > >> Airflow with Celery executor and run Kubernetes pods from Airflow > > using > > > >> in_cluster mode. Not sure if it allows to deploy Kubernetes executor > > > >> though. > > > >> > > > >> On Mon, Oct 7, 2019 at 8:43 AM Jarek Potiuk < > [email protected] > > > > > > >> wrote: > > > >> > > > >> > I just created "AIP-26 - Production-ready Airflow Docker Image and > > > helm > > > >> > chart > > > >> > < > > > >> > > > > >> > > > > > > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-26+-+Production-ready+Airflow+Docker+Image+and+helm+chart > > > >> > >" > > > >> > proposal and would love community feedback and comments. > > > >> > > > > >> > I would love to know what is your expectation from the Docker > image > > > and > > > >> how > > > >> > you are using production images currently. > > > >> > > > > >> > We were looking at the images available together with Daniel > > Imberman > > > >> > (including puckel and Astronomer image) and I started to work on a > > > draft > > > >> > POC > > > >> > <https://github.com/apache/airflow/pull/6266> to incorporate > > building > > > >> of > > > >> > the production-ready image into the CI framework we have. The > > current > > > >> POC > > > >> > uses the same Dockerfile we used for CI images but run with > > > >> > production-ready parameters. I made a number of refactorings and > > > >> > simplifications to the build logic and the scripts should be much > > more > > > >> sane > > > >> > now (and unit-testable soon). > > > >> > > > > >> > The images are based on Debian buster (Debian 10) LTS released few > > > >> months > > > >> > ago (so far we used Debian stretch for CI). It still needs some > > fixes > > > >> for > > > >> > failing tests ( > > https://travis-ci.org/potiuk/airflow/builds/594281768) > > > >> but > > > >> > the image itself can be downloaded from my private repo: `docker > > pull > > > >> > potiuk/airflow:master-python3.5`) > > > >> > > > > >> > The Docker file is available here: > > > >> > > > https://github.com/PolideaInternal/airflow/blob/prod-image/Dockerfile > > > >> > > > > >> > Once we have good image we will work with Daniel on helm chart so > > that > > > >> > Airflow can be easily installed on Kubernetes. > > > >> > > > > >> > Any comments and feedback/discussion in the AIP-26 document are > > > welcome > > > >> ! > > > >> > > > > >> > J. > > > >> > > > > >> > -- > > > >> > > > > >> > Jarek Potiuk > > > >> > Polidea <https://www.polidea.com/> | Principal Software Engineer > > > >> > > > > >> > M: +48 660 796 129 <+48660796129> > > > >> > [image: Polidea] <https://www.polidea.com/> > > > >> > > > > >> > > > > > > > > > > > > -- > > > > > > > > Jarek Potiuk > > > > Polidea <https://www.polidea.com/> | Principal Software Engineer > > > > > > > > M: +48 660 796 129 <+48660796129> > > > > [image: Polidea] <https://www.polidea.com/> > > > > > > > > > > > > > > -- > > > > > > Jarek Potiuk > > > Polidea <https://www.polidea.com/> | Principal Software Engineer > > > > > > M: +48 660 796 129 <+48660796129> > > > [image: Polidea] <https://www.polidea.com/> > > > > > > > > -- > > Jarek Potiuk > Polidea <https://www.polidea.com/> | Principal Software Engineer > > M: +48 660 796 129 <+48660796129> > [image: Polidea] <https://www.polidea.com/> >
