Hey Greg, I just wanted to update you with some recent changes I've done to automate most of the image builds stuff we have in Airflow. That might be useful for your efforts.
I am adding (and I will fix some of them very soon) issues based on feedback from people on the #prod-docker-image channel: https://github.com/apache/airflow/issues?q=is%3Aopen+is%3Aissue+label%3Aarea%3Aproduction-image But in the meantime all the build/CI infrastructure has nicely fallen into place and It's almost complete (one thing missing but very simple is automated builds of release images - for now this is manual but it is super easy change and I will add it before 1.10.11 for sure). I described the whole build infrastructure here: https://github.com/apache/airflow/blob/master/CI.rst including how image caching works, how nightly builds and images are triggered. I hope it will be helpful in the helm-chart work. Any questions or asks for the image - please let me know, it is super easy to update the image now when we have infrastructure to build it automatically integrated well with Github Actions and DockerHub. J. On Fri, Apr 10, 2020 at 7:23 PM Jarek Potiuk <jarek.pot...@polidea.com> wrote: > Awesome, thanks for the update! > > On Fri, Apr 10, 2020 at 6:57 PM Greg Neiheisel <g...@astronomer.io> wrote: > >> Hey everyone, just wanted to post a quick update on the helm chart. We're >> in the process of removing any references to Astronomer specific things and >> making them more general features. That work is happening on this PR here >> https://github.com/astronomer/airflow-chart/pull/66. Hoping to wrap up >> the last few outstanding items and open up a PR to Airflow next week >> sometime! >> >> On Fri, Apr 3, 2020 at 10:19 AM Daniel Imberman < >> daniel.imber...@gmail.com> wrote: >> >>> Hi Jarek, >>> >>> Congrats on getting the image out! This is going to make a lot of things >>> easier :). We’ve actually been working on separating out KinD as well (and >>> in fact moving it out of travis and into GitHub issues). Let me circle back >>> with Steven (the engineer who’s been working on that) and get a status >>> report from him. >>> >>> I will also take some time today to get our helm chart to work with >>> the airflow master image. It’s been a bit crazy around here in the last few >>> weeks but I am consistently pushing to make sure we make the final changes >>> of ripping out any remaining platform components before we donate :). >>> >>> I agree that docker hub automation will be great. This will also allow >>> us to have a nightly build for any integration/UI tests we build. >>> >>> Thanks again Jarek! >>> >>> Daniel >>> >>> via Newton Mail >>> <https://cloudmagic.com/k/d/mailapp?ct=dx&cv=10.0.32&pv=10.14.6&source=email_footer_2> >>> >>> On Fri, Apr 3, 2020 at 1:04 AM, Jarek Potiuk <jarek.pot...@polidea.com> >>> wrote: >>> >>> Hello Greg and Daniel, >>> >>> Production image is merged and it's also going to be released in >>> 1.10.10. For now, I will manually tag the release in DockerHub when we >>> release but we will automate that in the future. Currently, we have >>> *airflow:master* and *airflow:v1-10-test* images (for all supported >>> python versions) that you can use for testing with the Helm chart! >>> >>> The next things from my side: >>> >>> * I will automate DockerHub building for the prod image from master - >>> for now only CI is automated but we need to add prod image as well >>> * I will move kind-cluster creation outside of the container and use >>> production image for our CI Kubernetes tests. I have a WIP branch on it and >>> I think I should have it early next week >>> * The next step will be to switch our CI to Github Actions finally. I >>> think we will run both Travis and GA for a short time to make sure it works >>> * I will ask people for options they would like to have in the image - >>> additional packages etc. so that I can work on the next iteration of the >>> image. Also if you have any needs to see for the Helm integration, I am >>> super happy to incorporate them in the next version. I plan to keep the >>> "master" and "v1-10-test" images updated with any new changes and push >>> images regularly >>> * I am happy to help with the Helm Chart, so let me know if you need >>> anything! >>> >>> J. >>> >>> >>> On Mon, Mar 30, 2020 at 11:21 PM Jarek Potiuk <jarek.pot...@polidea.com> >>> wrote: >>> >>>> Hello Daniel and Greg. >>>> >>>> I think the production image for Airflow is quite ready for the final >>>> review and merge. I did some tests and I think it's quite complete now. >>>> >>>> The PR is running it's Travis build: >>>> https://github.com/apache/airflow/pull/7832 >>>> >>>> I already pushed the image to DockerHub - so that the next time it is >>>> merged It will use it as a base and will be rather quickly built using >>>> those images as a cache. Once it is merged, I will add the automated build >>>> in DockerHub and merge it to 1.10. As the next step I also plan to use the >>>> prod image for our Kubernetes tests (that will help with moving to the >>>> GitHub Actions from Travis). >>>> >>>> I added quite a comprehensive documentation - explaining all ins- and >>>> outs- of both CI and PROD images >>>> https://github.com/PolideaInternal/airflow/blob/add-production-docker-image/IMAGES.rst >>>> . >>>> >>>> You can build it yourself using manual build process described in the >>>> doc or (easier) using Breeze. That includes using the same Dockerfile to >>>> build older Airflow version from the 1.10.* line. >>>> >>>> Feel free to use the new image in the Helm Chart you have - happy to >>>> review the PRs. The image is available at 'apache/airflow:master-python3.6' >>>> and 'apache/airflow:master-python3.7` - once we merge it to 1.10 and >>>> release 1.10.10 we will also have it automatically published for 1.10.10 >>>> and beyond. >>>> >>>> Happy to hear your review comments and hope to merge it very soon. >>>> J. >>>> >>>> On Thu, Mar 26, 2020 at 4:38 PM Jarek Potiuk <jarek.pot...@polidea.com> >>>> wrote: >>>> >>>>> Absolutely! Please do :). The more eyes on this the better! >>>>> >>>>> On Thu, Mar 26, 2020 at 4:32 PM Daniel Imberman < >>>>> daniel.imber...@gmail.com> wrote: >>>>> >>>>>> @Jarek Steven and I will look into getting KinD to work on GitHub >>>>>> actions today. If we can get that working, we can donate the helm chart >>>>>> and >>>>>> possibly even deprecate the old Kubernetes tests in the process. >>>>>> On Mar 26, 2020, 8:27 AM -0700, Jarek Potiuk < >>>>>> jarek.pot...@polidea.com>, wrote: >>>>>> > We have A POC for Github Actions done by Tomek, and the blocker >>>>>> there >>>>>> > was precisely Kubernetes tests and specifically running kind cluster >>>>>> > in the Github Actions >>>>>> > https://github.com/PolideaInternal/airflow/pull/542 . In the >>>>>> meantime >>>>>> > Jiaje Zhong is trying to make it works on GA we discussed this very >>>>>> > thing today at slack: >>>>>> > >>>>>> https://apache-airflow.slack.com/archives/CCPRP7943/p1585073249374200 >>>>>> > >>>>>> > For me this is the next thing I will look at after merging >>>>>> > requirements PR and Prod image PR (and those two are a bit >>>>>> > pre-requisites to make it stable and fast). I want to switch to the >>>>>> > Production Image to run Kubernetes tests (they will be much faster) >>>>>> > and then we can work out much faster how to run them on Github >>>>>> > Actions. Then we can add Helm charts to run all those different >>>>>> > deployments. I am really looking forward to it! >>>>>> > >>>>>> > I hope - this weekend I move forward all of that and have a PR that >>>>>> we >>>>>> > can start running tests on and replacing the resources with helm >>>>>> > chart. >>>>>> > >>>>>> > J. >>>>>> > >>>>>> > On Thu, Mar 26, 2020 at 4:20 PM Daniel Imberman >>>>>> > <daniel.imber...@gmail.com> wrote: >>>>>> > > >>>>>> > > @Steven glad to help you out on that. >>>>>> > > >>>>>> > > For the moment we only use travis, but I believe @jarek has been >>>>>> looking to move to GitHub CI/CD. Perhaps we can use this as our first >>>>>> GitHub tests? >>>>>> > > On Mar 26, 2020, 8:17 AM -0700, Steven Miller >>>>>> <ste...@astronomer.io.invalid>, wrote: >>>>>> > > > Hey I’m happy to PR in the chart and get the CI working to test >>>>>> it. Where >>>>>> > > > do we want to run it? Do you all just use Travis only? >>>>>> > > > >>>>>> > > > Steven >>>>>> > > > >>>>>> > > > On Thu, Mar 26, 2020 at 10:59 AM Daniel Imberman < >>>>>> daniel.imber...@gmail.com> >>>>>> > > > wrote: >>>>>> > > > >>>>>> > > > > @Jarek I think with the helm chart + prod image we can go >>>>>> even further >>>>>> > > > > than that :). We can test CeleryExecutor, with KEDA >>>>>> autoscaling, and a >>>>>> > > > > bunch of other configurations. >>>>>> > > > > On Mar 26, 2020, 7:45 AM -0700, Jarek Potiuk < >>>>>> jarek.pot...@polidea.com>, >>>>>> > > > > wrote: >>>>>> > > > > > Yeah. I meant Custom Resources not CRDs in my original >>>>>> email :) >>>>>> > > > > > >>>>>> > > > > > > On Thu, Mar 26, 2020 at 3:38 PM Daniel Imberman < >>>>>> > > > > daniel.imber...@gmail.com> wrote: >>>>>> > > > > > > > We’re not using CRDs for the tests at the moment. We >>>>>> just have >>>>>> > > > > deployment files. If anything having the helm chart as a part >>>>>> of the >>>>>> > > > > airflow repo could mean that the helm chart becomes the >>>>>> defacto system for >>>>>> > > > > testing airflow on kubernetes (we can get rid of all the yams >>>>>> files and run >>>>>> > > > > multiple k8s tests with different settings). >>>>>> > > > > > > > On Mar 26, 2020, 7:20 AM -0700, Greg Neiheisel >>>>>> > > > > <g...@astronomer.io.invalid>, wrote: >>>>>> > > > > > > > > Yep, we can absolutely pull it into the airflow repo. >>>>>> We've also >>>>>> > > > > been >>>>>> > > > > > > > > building up a test suite that currently runs on >>>>>> CircleCI and uses >>>>>> > > > > kind >>>>>> > > > > > > > > (Kubernetes in Docker) to test several kubernetes >>>>>> versions with >>>>>> > > > > some >>>>>> > > > > > > > > different settings. Right now we're mostly testing >>>>>> the different >>>>>> > > > > executors >>>>>> > > > > > > > > since that has the biggest impact on what gets >>>>>> deployed, but that >>>>>> > > > > can be >>>>>> > > > > > > > > expanded. >>>>>> > > > > > > > > >>>>>> > > > > > > > > What CRDs are currently being used to run Airflow for >>>>>> the tests? >>>>>> > > > > > > > > >>>>>> > > > > > > > > On Wed, Mar 25, 2020 at 11:06 AM Jarek Potiuk < >>>>>> > > > > jarek.pot...@polidea.com> >>>>>> > > > > > > > > wrote: >>>>>> > > > > > > > > >>>>>> > > > > > > > > > One thing for the donation. >>>>>> > > > > > > > > > >>>>>> > > > > > > > > > Did you you want to have separate repository Greg ? >>>>>> > > > > > > > > > >>>>>> > > > > > > > > > I think we should simply create a folder in Airflow >>>>>> repo and >>>>>> > > > > keep it >>>>>> > > > > > > > > > there (similarly as we keep Dockerfile). I am going >>>>>> to switch our >>>>>> > > > > > > > > > Kubernetes Tests to the production image (will make >>>>>> the tests >>>>>> > > > > much >>>>>> > > > > > > > > > faster) and I am going to test the Dockerfile >>>>>> automatically in >>>>>> > > > > CI - >>>>>> > > > > > > > > > for now we are using some custom Resource >>>>>> definitions to start >>>>>> > > > > Airflow >>>>>> > > > > > > > > > on Kubernetes Cluster for the tests, but we could >>>>>> switch to >>>>>> > > > > using the >>>>>> > > > > > > > > > helm chart - this way we can test all three things >>>>>> at once: >>>>>> > > > > > > > > > - Kubernetes Executor >>>>>> > > > > > > > > > - Dockerfile >>>>>> > > > > > > > > > - Helm Chart >>>>>> > > > > > > > > > and we could also add more tests - for example >>>>>> testing different >>>>>> > > > > > > > > > deployment options for the helm chart. >>>>>> > > > > > > > > > >>>>>> > > > > > > > > > Having the Helm chart in Airflow repo would help >>>>>> with that - >>>>>> > > > > > > > > > especially in terms of future changes and testing >>>>>> them >>>>>> > > > > automatically. >>>>>> > > > > > > > > > >>>>>> > > > > > > > > > J. >>>>>> > > > > > > > > > >>>>>> > > > > > > > > > On Tue, Mar 24, 2020 at 9:09 PM Aizhamal Nurmamat >>>>>> kyzy >>>>>> > > > > > > > > > <aizha...@apache.org> wrote: >>>>>> > > > > > > > > > > >>>>>> > > > > > > > > > > +1 on the donation. Always happy to see more >>>>>> useful stuff for >>>>>> > > > > the >>>>>> > > > > > > > > > > community :) >>>>>> > > > > > > > > > > >>>>>> > > > > > > > > > > On Tue, Mar 24, 2020 at 9:20 AM Greg Neiheisel < >>>>>> > > > > g...@astronomer.io> >>>>>> > > > > > > > > > wrote: >>>>>> > > > > > > > > > > >>>>>> > > > > > > > > > > > Yep, the cleanup_pods script is set up now as >>>>>> an optional >>>>>> > > > > Kubernetes >>>>>> > > > > > > > > > > > CronJob ( >>>>>> > > > > > > > > > > > >>>>>> > > > > > > > > > >>>>>> > > > > >>>>>> https://github.com/astronomer/airflow-chart/blob/master/templates/cleanup/cleanup-cronjob.yaml >>>>>> > > > > > > > > > ) >>>>>> > > > > > > > > > > > that we have run periodically to clean failed >>>>>> pods up and >>>>>> > > > > could stay >>>>>> > > > > > > > > > > > separate. >>>>>> > > > > > > > > > > > >>>>>> > > > > > > > > > > > The wait_for_migrations script could definitely >>>>>> be pulled >>>>>> > > > > into Airflow. >>>>>> > > > > > > > > > > > For context, we deploy an initContainer on the >>>>>> scheduler ( >>>>>> > > > > > > > > > > > >>>>>> > > > > > > > > > >>>>>> > > > > >>>>>> https://github.com/astronomer/airflow-chart/blob/master/templates/scheduler/scheduler-deployment.yaml#L77-L84 >>>>>> > > > > > > > > > ) >>>>>> > > > > > > > > > > > that runs the upgradedb command before booting >>>>>> the >>>>>> > > > > scheduler. This new >>>>>> > > > > > > > > > > > wait_for_migration script runs in an >>>>>> initContainer on the >>>>>> > > > > webserver and >>>>>> > > > > > > > > > > > workers ( >>>>>> > > > > > > > > > > > >>>>>> > > > > > > > > > >>>>>> > > > > >>>>>> https://github.com/astronomer/airflow-chart/blob/master/templates/webserver/webserver-deployment.yaml#L58-L65 >>>>>> > > > > > > > > > ) >>>>>> > > > > > > > > > > > so that they don't boot up ahead of a >>>>>> potentially >>>>>> > > > > long-running >>>>>> > > > > > > > > > migration >>>>>> > > > > > > > > > > > and attempt to operate on new or missing >>>>>> columns/tables >>>>>> > > > > before the >>>>>> > > > > > > > > > > > migrations run. This prevents these pods from >>>>>> entering a >>>>>> > > > > CrashLoop. >>>>>> > > > > > > > > > > > >>>>>> > > > > > > > > > > > On Tue, Mar 24, 2020 at 11:48 AM Jarek Potiuk < >>>>>> > > > > > > > > > jarek.pot...@polidea.com> >>>>>> > > > > > > > > > > > wrote: >>>>>> > > > > > > > > > > > >>>>>> > > > > > > > > > > > > > >>>>>> > > > > > > > > > > > > > @Tomasz great question. Our images are >>>>>> currently >>>>>> > > > > generated from >>>>>> > > > > > > > > > > > > Dockerfiles >>>>>> > > > > > > > > > > > > > in this repo >>>>>> https://github.com/astronomer/ap-airflow >>>>>> > > > > and get >>>>>> > > > > > > > > > > > > published to >>>>>> > > > > > > > > > > > > > DockerHub >>>>>> > > > > > > > > > > > > > >>>>>> > > > > >>>>>> https://hub.docker.com/repository/docker/astronomerinc/ap-airflow. >>>>>> > > > > > > > > > > > > > >>>>>> > > > > > > > > > > > > > For the most part those are typical Airflow >>>>>> images. >>>>>> > > > > There's an >>>>>> > > > > > > > > > > > > entrypoint >>>>>> > > > > > > > > > > > > > script that we include in the image that >>>>>> handles waiting >>>>>> > > > > for the >>>>>> > > > > > > > > > > > > database >>>>>> > > > > > > > > > > > > > and redis (if used) to come up, which is >>>>>> pretty generic. >>>>>> > > > > > > > > > > > > >>>>>> > > > > > > > > > > > > >>>>>> > > > > > > > > > > > > I already added waiting for the database >>>>>> (both metadata >>>>>> > > > > and celery >>>>>> > > > > > > > > > URL) in >>>>>> > > > > > > > > > > > > the PR: >>>>>> > > > > > > > > > > > > >>>>>> > > > > > > > > > > > > >>>>>> > > > > > > > > > >>>>>> > > > > >>>>>> https://github.com/apache/airflow/pull/7832/files#diff-3759f40d4e8ba0c0e82e82b66d376741 >>>>>> > > > > > > > > > > > > . >>>>>> > > > > > > > > > > > > It's functionally the same but more generic. >>>>>> > > > > > > > > > > > > >>>>>> > > > > > > > > > > > > The only other >>>>>> > > > > > > > > > > > > > thing that I think the Helm Chart uses >>>>>> would be the >>>>>> > > > > scripts in this >>>>>> > > > > > > > > > repo >>>>>> > > > > > > > > > > > > > >>>>>> https://github.com/astronomer/astronomer-airflow-scripts. >>>>>> > > > > Our >>>>>> > > > > > > > > > > > > Dockerfiles >>>>>> > > > > > > > > > > > > > pull this package in. These scripts are >>>>>> used to >>>>>> > > > > coordinate running >>>>>> > > > > > > > > > > > > > migrations and cleaning up failed pods. >>>>>> > > > > > > > > > > > > > >>>>>> > > > > > > > > > > > > >>>>>> > > > > > > > > > > > > I see two scripts: >>>>>> > > > > > > > > > > > > >>>>>> > > > > > > > > > > > > * cleanup_pods -> this is (I believe) not >>>>>> needed to run in >>>>>> > > > > airflow - >>>>>> > > > > > > > > > this >>>>>> > > > > > > > > > > > > could be run as a separate pod/container? >>>>>> > > > > > > > > > > > > * waiting for migrations -> I think this is a >>>>>> good >>>>>> > > > > candidate to add >>>>>> > > > > > > > > > > > > *airflow >>>>>> > > > > > > > > > > > > db wait_for_migration* command and make it >>>>>> part of airflow >>>>>> > > > > itself. >>>>>> > > > > > > > > > > > > >>>>>> > > > > > > > > > > > > I think we also have to agree on the Airflow >>>>>> version >>>>>> > > > > supported by the >>>>>> > > > > > > > > > > > > official helm chart. I'd suggest we support >>>>>> 1.10.10+ and we >>>>>> > > > > > > > > > incorporate >>>>>> > > > > > > > > > > > > all >>>>>> > > > > > > > > > > > > the changes needed to airflow (like the "db >>>>>> > > > > wait_for_migration") >>>>>> > > > > > > > > > into 2.0 >>>>>> > > > > > > > > > > > > and 1.10 and we support both - image and helm >>>>>> chart for >>>>>> > > > > those versions >>>>>> > > > > > > > > > > > > only. That would help with people migrating >>>>>> to the latest >>>>>> > > > > version. >>>>>> > > > > > > > > > > > > >>>>>> > > > > > > > > > > > > WDYT? >>>>>> > > > > > > > > > > > > >>>>>> > > > > > > > > > > > > >>>>>> > > > > > > > > > > > > > On Tue, Mar 24, 2020 at 10:49 AM Daniel >>>>>> Imberman < >>>>>> > > > > > > > > > > > > > daniel.imber...@gmail.com> >>>>>> > > > > > > > > > > > > > wrote: >>>>>> > > > > > > > > > > > > > >>>>>> > > > > > > > > > > > > > > @jarek I agree completely. I think that >>>>>> pairing an >>>>>> > > > > official helm >>>>>> > > > > > > > > > chart >>>>>> > > > > > > > > > > > > > > with the official image would make for a >>>>>> REALLY >>>>>> > > > > powerful “up and >>>>>> > > > > > > > > > > > > running >>>>>> > > > > > > > > > > > > > > with airflow” story :). Tomek and I have >>>>>> also been >>>>>> > > > > looking into >>>>>> > > > > > > > > > > > > > > operator-sdk which has the ability to >>>>>> create custom >>>>>> > > > > controllers >>>>>> > > > > > > > > > from >>>>>> > > > > > > > > > > > > helm >>>>>> > > > > > > > > > > > > > > charts. We might even able to get a 1-2 >>>>>> punch from the >>>>>> > > > > same code >>>>>> > > > > > > > > > base >>>>>> > > > > > > > > > > > > :). >>>>>> > > > > > > > > > > > > > > >>>>>> > > > > > > > > > > > > > > @kaxil @jarek @aizhamal @ash if there’s >>>>>> no issues, can >>>>>> > > > > we please >>>>>> > > > > > > > > > start >>>>>> > > > > > > > > > > > > > the >>>>>> > > > > > > > > > > > > > > process of donation? >>>>>> > > > > > > > > > > > > > > >>>>>> > > > > > > > > > > > > > > +1 on my part, of course :) >>>>>> > > > > > > > > > > > > > > >>>>>> > > > > > > > > > > > > > > >>>>>> > > > > > > > > > > > > > > >>>>>> > > > > > > > > > > > > > > Daniel >>>>>> > > > > > > > > > > > > > > On Mar 24, 2020, 7:40 AM -0700, Jarek >>>>>> Potiuk < >>>>>> > > > > > > > > > > > > jarek.pot...@polidea.com>, >>>>>> > > > > > > > > > > > > > > wrote: >>>>>> > > > > > > > > > > > > > > > +1. And it should be paired with the >>>>>> official image >>>>>> > > > > we have >>>>>> > > > > > > > > > work in >>>>>> > > > > > > > > > > > > > > > progress on. I looked a lot at the >>>>>> Astronomer's >>>>>> > > > > image while >>>>>> > > > > > > > > > > > > preparing >>>>>> > > > > > > > > > > > > > my >>>>>> > > > > > > > > > > > > > > > draft and we can make any adjustments >>>>>> needed to make >>>>>> > > > > it works >>>>>> > > > > > > > > > with >>>>>> > > > > > > > > > > > > the >>>>>> > > > > > > > > > > > > > > helm >>>>>> > > > > > > > > > > > > > > > chart - and I am super happy to >>>>>> collaborate on that. >>>>>> > > > > > > > > > > > > > > > >>>>>> > > > > > > > > > > > > > > > PR here: >>>>>> https://github.com/apache/airflow/pull/7832 >>>>>> > > > > > > > > > > > > > > > >>>>>> > > > > > > > > > > > > > > > J. >>>>>> > > > > > > > > > > > > > > > >>>>>> > > > > > > > > > > > > > > > >>>>>> > > > > > > > > > > > > > > > On Tue, Mar 24, 2020 at 3:15 PM Kaxil >>>>>> Naik < >>>>>> > > > > kaxiln...@gmail.com >>>>>> > > > > > > > > > > >>>>>> > > > > > > > > > > > > > wrote: >>>>>> > > > > > > > > > > > > > > > >>>>>> > > > > > > > > > > > > > > > > @Tomasz Urbaszek < >>>>>> tomasz.urbas...@polidea.com> : >>>>>> > > > > > > > > > > > > > > > > Helm Chart Link: >>>>>> > > > > https://github.com/astronomer/airflow-chart >>>>>> > > > > > > > > > > > > > > > > >>>>>> > > > > > > > > > > > > > > > > On Tue, Mar 24, 2020 at 2:13 PM >>>>>> Tomasz Urbaszek < >>>>>> > > > > > > > > > > > > > turbas...@apache.org> >>>>>> > > > > > > > > > > > > > > > > wrote: >>>>>> > > > > > > > > > > > > > > > > >>>>>> > > > > > > > > > > > > > > > > > An official helm chart is something >>>>>> our >>>>>> > > > > community needs! >>>>>> > > > > > > > > > Using >>>>>> > > > > > > > > > > > > your >>>>>> > > > > > > > > > > > > > > > > > chart as the official makes a lot >>>>>> of sens to me >>>>>> > > > > because as >>>>>> > > > > > > > > > you >>>>>> > > > > > > > > > > > > > > > > > mentioned - it's battle tested. >>>>>> > > > > > > > > > > > > > > > > > >>>>>> > > > > > > > > > > > > > > > > > One question: what Airflow image do >>>>>> you use? >>>>>> > > > > Also, would you >>>>>> > > > > > > > > > > > > mind >>>>>> > > > > > > > > > > > > > > > > > sharing a link to the chart? >>>>>> > > > > > > > > > > > > > > > > > >>>>>> > > > > > > > > > > > > > > > > > Tomek >>>>>> > > > > > > > > > > > > > > > > > >>>>>> > > > > > > > > > > > > > > > > > >>>>>> > > > > > > > > > > > > > > > > > On Tue, Mar 24, 2020 at 2:07 PM >>>>>> Greg Neiheisel >>>>>> > > > > > > > > > > > > > > > > > <g...@astronomer.io.invalid> wrote: >>>>>> > > > > > > > > > > > > > > > > > > >>>>>> > > > > > > > > > > > > > > > > > > Hey everyone, >>>>>> > > > > > > > > > > > > > > > > > > >>>>>> > > > > > > > > > > > > > > > > > > Over the past few years at >>>>>> Astronomer, we’ve >>>>>> > > > > created, >>>>>> > > > > > > > > > managed, >>>>>> > > > > > > > > > > > > > and >>>>>> > > > > > > > > > > > > > > > > > hardened >>>>>> > > > > > > > > > > > > > > > > > > a production-ready Helm Chart for >>>>>> Airflow ( >>>>>> > > > > > > > > > > > > > > > > > > >>>>>> https://github.com/astronomer/airflow-chart) >>>>>> > > > > that is >>>>>> > > > > > > > > > being >>>>>> > > > > > > > > > > > > used >>>>>> > > > > > > > > > > > > > by >>>>>> > > > > > > > > > > > > > > > > both >>>>>> > > > > > > > > > > > > > > > > > our >>>>>> > > > > > > > > > > > > > > > > > > SaaS and Enterprise customers. >>>>>> This chart is >>>>>> > > > > > > > > > battle-tested and >>>>>> > > > > > > > > > > > > > > running >>>>>> > > > > > > > > > > > > > > > > > > hundreds of Airflow deployments >>>>>> of varying >>>>>> > > > > sizes and >>>>>> > > > > > > > > > runtime >>>>>> > > > > > > > > > > > > > > > > > environments. >>>>>> > > > > > > > > > > > > > > > > > > It’s been built up to encapsulate >>>>>> the issues >>>>>> > > > > that Airflow >>>>>> > > > > > > > > > > > > users >>>>>> > > > > > > > > > > > > > run >>>>>> > > > > > > > > > > > > > > > > into >>>>>> > > > > > > > > > > > > > > > > > in >>>>>> > > > > > > > > > > > > > > > > > > the real world. >>>>>> > > > > > > > > > > > > > > > > > > >>>>>> > > > > > > > > > > > > > > > > > > While this chart was originally >>>>>> developed >>>>>> > > > > internally for >>>>>> > > > > > > > > > our >>>>>> > > > > > > > > > > > > > > Astronomer >>>>>> > > > > > > > > > > > > > > > > > > Platform, we’ve recently >>>>>> decoupled the chart >>>>>> > > > > from the >>>>>> > > > > > > > > > rest of >>>>>> > > > > > > > > > > > > our >>>>>> > > > > > > > > > > > > > > > > > platform >>>>>> > > > > > > > > > > > > > > > > > > to make it usable by the greater >>>>>> Airflow >>>>>> > > > > community. With >>>>>> > > > > > > > > > these >>>>>> > > > > > > > > > > > > > > changes >>>>>> > > > > > > > > > > > > > > > > in >>>>>> > > > > > > > > > > > > > > > > > > mind, we want to start a >>>>>> conversation about >>>>>> > > > > donating this >>>>>> > > > > > > > > > > > > chart >>>>>> > > > > > > > > > > > > > to >>>>>> > > > > > > > > > > > > > > the >>>>>> > > > > > > > > > > > > > > > > > > Airflow community. >>>>>> > > > > > > > > > > > > > > > > > > >>>>>> > > > > > > > > > > > > > > > > > > Some of the main features of the >>>>>> chart are: >>>>>> > > > > > > > > > > > > > > > > > > >>>>>> > > > > > > > > > > > > > > > > > > - It works out of the box. With >>>>>> zero >>>>>> > > > > configuration, a user >>>>>> > > > > > > > > > > > > will >>>>>> > > > > > > > > > > > > > get >>>>>> > > > > > > > > > > > > > > > > a >>>>>> > > > > > > > > > > > > > > > > > > postgres database, a default user >>>>>> and the >>>>>> > > > > > > > > > KubernetesExecutor >>>>>> > > > > > > > > > > > > > ready >>>>>> > > > > > > > > > > > > > > > > to >>>>>> > > > > > > > > > > > > > > > > > run >>>>>> > > > > > > > > > > > > > > > > > > DAGs. >>>>>> > > > > > > > > > > > > > > > > > > - Support for Local, Celery (w/ >>>>>> optional KEDA >>>>>> > > > > > > > > > autoscaling) and >>>>>> > > > > > > > > > > > > > > > > > > Kubernetes executors. >>>>>> > > > > > > > > > > > > > > > > > > >>>>>> > > > > > > > > > > > > > > > > > > Support for optional pgbouncer. >>>>>> We use this to >>>>>> > > > > share a >>>>>> > > > > > > > > > > > > > configurable >>>>>> > > > > > > > > > > > > > > > > > > connection pool size per >>>>>> deployment. Useful >>>>>> > > > > for limiting >>>>>> > > > > > > > > > > > > > > connections to >>>>>> > > > > > > > > > > > > > > > > > the >>>>>> > > > > > > > > > > > > > > > > > > metadata database. >>>>>> > > > > > > > > > > > > > > > > > > >>>>>> > > > > > > > > > > > > > > > > > > - Airflow migration support. A >>>>>> user can push a >>>>>> > > > > newer >>>>>> > > > > > > > > > version >>>>>> > > > > > > > > > > > > of >>>>>> > > > > > > > > > > > > > > > > > Airflow >>>>>> > > > > > > > > > > > > > > > > > > into an existing release and >>>>>> migrations will >>>>>> > > > > > > > > > automatically run >>>>>> > > > > > > > > > > > > > > > > > cleanly. >>>>>> > > > > > > > > > > > > > > > > > > - Prometheus support. Optionally >>>>>> install and >>>>>> > > > > configure a >>>>>> > > > > > > > > > > > > > > > > > statsd-exporter >>>>>> > > > > > > > > > > > > > > > > > > to ingest Airflow metrics and >>>>>> expose them to >>>>>> > > > > Prometheus >>>>>> > > > > > > > > > > > > > > > > automatically. >>>>>> > > > > > > > > > > > > > > > > > > - Resource control. Optionally >>>>>> control the >>>>>> > > > > ResourceQuotas >>>>>> > > > > > > > > > and >>>>>> > > > > > > > > > > > > > > > > > > LimitRanges for each deployment >>>>>> so that no >>>>>> > > > > deployment can >>>>>> > > > > > > > > > > > > > overload >>>>>> > > > > > > > > > > > > > > a >>>>>> > > > > > > > > > > > > > > > > > > cluster. >>>>>> > > > > > > > > > > > > > > > > > > - Simple optional Elasticsearch >>>>>> support. >>>>>> > > > > > > > > > > > > > > > > > > - Optional namespace cleanup. >>>>>> Sometimes >>>>>> > > > > > > > > > KubernetesExecutor and >>>>>> > > > > > > > > > > > > > > > > > > KubernetesPodOperator pods fail >>>>>> for reasons >>>>>> > > > > other than the >>>>>> > > > > > > > > > > > > actual >>>>>> > > > > > > > > > > > > > > > > > task. >>>>>> > > > > > > > > > > > > > > > > > > This feature helps keep things >>>>>> clean in >>>>>> > > > > Kubernetes. >>>>>> > > > > > > > > > > > > > > > > > > - Support for running locally in >>>>>> KIND >>>>>> > > > > (Kubernetes in >>>>>> > > > > > > > > > Docker). >>>>>> > > > > > > > > > > > > > > > > > > - Automatically tested across >>>>>> many Kubernetes >>>>>> > > > > versions >>>>>> > > > > > > > > > with >>>>>> > > > > > > > > > > > > Helm >>>>>> > > > > > > > > > > > > > 2 >>>>>> > > > > > > > > > > > > > > > > > and 3 >>>>>> > > > > > > > > > > > > > > > > > > support. >>>>>> > > > > > > > > > > > > > > > > > > >>>>>> > > > > > > > > > > > > > > > > > > We’ve found that the cleanest and >>>>>> most >>>>>> > > > > reliable way to >>>>>> > > > > > > > > > deploy >>>>>> > > > > > > > > > > > > > DAGs >>>>>> > > > > > > > > > > > > > > to >>>>>> > > > > > > > > > > > > > > > > > > Kubernetes and manage them at >>>>>> scale is to >>>>>> > > > > package them >>>>>> > > > > > > > > > into >>>>>> > > > > > > > > > > > > the >>>>>> > > > > > > > > > > > > > > actual >>>>>> > > > > > > > > > > > > > > > > > > docker image, so we have geared >>>>>> this chart >>>>>> > > > > towards that >>>>>> > > > > > > > > > > > > method of >>>>>> > > > > > > > > > > > > > > > > > > operation, though adding other >>>>>> methods should >>>>>> > > > > be >>>>>> > > > > > > > > > > > > straightforward. >>>>>> > > > > > > > > > > > > > > > > > > >>>>>> > > > > > > > > > > > > > > > > > > We would love thoughts from the >>>>>> community and >>>>>> > > > > would love >>>>>> > > > > > > > > > to >>>>>> > > > > > > > > > > > > see >>>>>> > > > > > > > > > > > > > > this >>>>>> > > > > > > > > > > > > > > > > > chart >>>>>> > > > > > > > > > > > > > > > > > > help others to get up and running >>>>>> on >>>>>> > > > > Kubernetes! >>>>>> > > > > > > > > > > > > > > > > > > >>>>>> > > > > > > > > > > > > > > > > > > -- >>>>>> > > > > > > > > > > > > > > > > > > *Greg Neiheisel* / Chief Architect >>>>>> > > > > Astronomer.io >>>>>> > > > > > > > > > > > > > > > > > >>>>>> > > > > > > > > > > > > > > > > >>>>>> > > > > > > > > > > > > > > > >>>>>> > > > > > > > > > > > > > > > >>>>>> > > > > > > > > > > > > > > > -- >>>>>> > > > > > > > > > > > > > > > >>>>>> > > > > > > > > > > > > > > > Jarek Potiuk >>>>>> > > > > > > > > > > > > > > > Polidea <https://www.polidea.com/> | >>>>>> Principal >>>>>> > > > > Software >>>>>> > > > > > > > > > Engineer >>>>>> > > > > > > > > > > > > > > > >>>>>> > > > > > > > > > > > > > > > M: +48 660 796 129 >>>>>> <+48%20660%20796%20129> >>>>>> > > > > <+48660796129 >>>>>> > > > > > > > > > > > > <+48%20660%20796%20129>> >>>>>> > > > > > > > > > > > > > > > [image: Polidea] < >>>>>> https://www.polidea.com/> >>>>>> > > > > > > > > > > > > > > >>>>>> > > > > > > > > > > > > > >>>>>> > > > > > > > > > > > > > >>>>>> > > > > > > > > > > > > > -- >>>>>> > > > > > > > > > > > > > *Greg Neiheisel* / Chief Architect >>>>>> Astronomer.io >>>>>> > > > > > > > > > > > > > >>>>>> > > > > > > > > > > > > >>>>>> > > > > > > > > > > > > >>>>>> > > > > > > > > > > > > -- >>>>>> > > > > > > > > > > > > >>>>>> > > > > > > > > > > > > Jarek Potiuk >>>>>> > > > > > > > > > > > > Polidea <https://www.polidea.com/> | >>>>>> Principal Software >>>>>> > > > > Engineer >>>>>> > > > > > > > > > > > > >>>>>> > > > > > > > > > > > > M: +48 660 796 129 <+48%20660%20796%20129> >>>>>> <+48660796129 >>>>>> > > > > > > > > > > > > <+48%20660%20796%20129>> >>>>>> > > > > > > > > > > > > [image: Polidea] <https://www.polidea.com/> >>>>>> > > > > > > > > > > > > >>>>>> > > > > > > > > > > > >>>>>> > > > > > > > > > > > >>>>>> > > > > > > > > > > > -- >>>>>> > > > > > > > > > > > *Greg Neiheisel* / Chief Architect Astronomer.io >>>>>> > > > > > > > > > > > >>>>>> > > > > > > > > > >>>>>> > > > > > > > > > >>>>>> > > > > > > > > > >>>>>> > > > > > > > > > -- >>>>>> > > > > > > > > > >>>>>> > > > > > > > > > Jarek Potiuk >>>>>> > > > > > > > > > Polidea | Principal Software Engineer >>>>>> > > > > > > > > > >>>>>> > > > > > > > > > M: +48 660 796 129 >>>>>> > > > > > > > > > >>>>>> > > > > > > > > >>>>>> > > > > > > > > >>>>>> > > > > > > > > -- >>>>>> > > > > > > > > *Greg Neiheisel* / Chief Architect Astronomer.io >>>>>> > > > > > >>>>>> > > > > > >>>>>> > > > > > -- >>>>>> > > > > > Jarek Potiuk >>>>>> > > > > > Polidea | Principal Software Engineer >>>>>> > > > > > M: +48 660 796 129 >>>>>> > > > > > >>>>>> > > > > >>>>>> > >>>>>> > >>>>>> > >>>>>> > -- >>>>>> > >>>>>> > Jarek Potiuk >>>>>> > Polidea | Principal Software Engineer >>>>>> > >>>>>> > M: +48 660 796 129 >>>>>> >>>>> >>>>> >>>>> -- >>>>> >>>>> Jarek Potiuk >>>>> Polidea <https://www.polidea.com/> | Principal Software Engineer >>>>> >>>>> M: +48 660 796 129 <+48660796129> >>>>> [image: Polidea] <https://www.polidea.com/> >>>>> >>>>> >>>> >>>> -- >>>> >>>> Jarek Potiuk >>>> Polidea <https://www.polidea.com/> | Principal Software Engineer >>>> >>>> M: +48 660 796 129 <+48660796129> >>>> [image: Polidea] <https://www.polidea.com/> >>>> >>>> >>> >>> -- >>> >>> Jarek Potiuk >>> Polidea <https://www.polidea.com/> | Principal Software Engineer >>> >>> M: +48 660 796 129 <+48660796129> >>> [image: Polidea] <https://www.polidea.com/> >>> >>> >> >> -- >> *Greg Neiheisel* / Chief Architect Astronomer.io >> > > > -- > > Jarek Potiuk > Polidea <https://www.polidea.com/> | Principal Software Engineer > > M: +48 660 796 129 <+48660796129> > [image: Polidea] <https://www.polidea.com/> > > -- Jarek Potiuk Polidea <https://www.polidea.com/> | Principal Software Engineer M: +48 660 796 129 <+48660796129> [image: Polidea] <https://www.polidea.com/>
