Awesome Jarek. I just opened up a draft PR that adds the chart into the Airflow repo. https://github.com/apache/airflow/pull/8777
It still contains some of our packaging, testing and releasing scripts. I figure they can serve as a starting point to discuss how we'd want to replicate into Airflow infrastructure. Looking forward to getting some feedback and moving this along! On Tue, Apr 28, 2020 at 8:00 AM Jarek Potiuk <jarek.pot...@polidea.com> wrote: > Hey Greg, > > I just wanted to update you with some recent changes I've done to automate > most of the image builds stuff we have in Airflow. That might be useful for > your efforts. > > I am adding (and I will fix some of them very soon) issues based on > feedback from people on the #prod-docker-image channel: > https://github.com/apache/airflow/issues?q=is%3Aopen+is%3Aissue+label%3Aarea%3Aproduction-image > But in the meantime all the build/CI infrastructure has nicely fallen into > place and It's almost complete (one thing missing but very simple is > automated builds of release images - for now this is manual but it is super > easy change and I will add it before 1.10.11 for sure). > > I described the whole build infrastructure here: > https://github.com/apache/airflow/blob/master/CI.rst including how image > caching works, how nightly builds and images are triggered. I hope it will > be helpful in the helm-chart work. Any questions or asks for the image - > please let me know, it is super easy to update the image now when we have > infrastructure to build it automatically integrated well with Github > Actions and DockerHub. > > J. > > > > > > > > > On Fri, Apr 10, 2020 at 7:23 PM Jarek Potiuk <jarek.pot...@polidea.com> > wrote: > >> Awesome, thanks for the update! >> >> On Fri, Apr 10, 2020 at 6:57 PM Greg Neiheisel <g...@astronomer.io> >> wrote: >> >>> Hey everyone, just wanted to post a quick update on the helm chart. >>> We're in the process of removing any references to Astronomer specific >>> things and making them more general features. That work is happening on >>> this PR here https://github.com/astronomer/airflow-chart/pull/66. >>> Hoping to wrap up the last few outstanding items and open up a PR to >>> Airflow next week sometime! >>> >>> On Fri, Apr 3, 2020 at 10:19 AM Daniel Imberman < >>> daniel.imber...@gmail.com> wrote: >>> >>>> Hi Jarek, >>>> >>>> Congrats on getting the image out! This is going to make a lot of >>>> things easier :). We’ve actually been working on separating out KinD as >>>> well (and in fact moving it out of travis and into GitHub issues). Let me >>>> circle back with Steven (the engineer who’s been working on that) and get a >>>> status report from him. >>>> >>>> I will also take some time today to get our helm chart to work with >>>> the airflow master image. It’s been a bit crazy around here in the last few >>>> weeks but I am consistently pushing to make sure we make the final changes >>>> of ripping out any remaining platform components before we donate :). >>>> >>>> I agree that docker hub automation will be great. This will also allow >>>> us to have a nightly build for any integration/UI tests we build. >>>> >>>> Thanks again Jarek! >>>> >>>> Daniel >>>> >>>> via Newton Mail >>>> <https://cloudmagic.com/k/d/mailapp?ct=dx&cv=10.0.32&pv=10.14.6&source=email_footer_2> >>>> >>>> On Fri, Apr 3, 2020 at 1:04 AM, Jarek Potiuk <jarek.pot...@polidea.com> >>>> wrote: >>>> >>>> Hello Greg and Daniel, >>>> >>>> Production image is merged and it's also going to be released in >>>> 1.10.10. For now, I will manually tag the release in DockerHub when we >>>> release but we will automate that in the future. Currently, we have >>>> *airflow:master* and *airflow:v1-10-test* images (for all supported >>>> python versions) that you can use for testing with the Helm chart! >>>> >>>> The next things from my side: >>>> >>>> * I will automate DockerHub building for the prod image from master - >>>> for now only CI is automated but we need to add prod image as well >>>> * I will move kind-cluster creation outside of the container and use >>>> production image for our CI Kubernetes tests. I have a WIP branch on it and >>>> I think I should have it early next week >>>> * The next step will be to switch our CI to Github Actions finally. I >>>> think we will run both Travis and GA for a short time to make sure it works >>>> * I will ask people for options they would like to have in the image - >>>> additional packages etc. so that I can work on the next iteration of the >>>> image. Also if you have any needs to see for the Helm integration, I am >>>> super happy to incorporate them in the next version. I plan to keep the >>>> "master" and "v1-10-test" images updated with any new changes and push >>>> images regularly >>>> * I am happy to help with the Helm Chart, so let me know if you need >>>> anything! >>>> >>>> J. >>>> >>>> >>>> On Mon, Mar 30, 2020 at 11:21 PM Jarek Potiuk <jarek.pot...@polidea.com> >>>> wrote: >>>> >>>>> Hello Daniel and Greg. >>>>> >>>>> I think the production image for Airflow is quite ready for the final >>>>> review and merge. I did some tests and I think it's quite complete now. >>>>> >>>>> The PR is running it's Travis build: >>>>> https://github.com/apache/airflow/pull/7832 >>>>> >>>>> I already pushed the image to DockerHub - so that the next time it is >>>>> merged It will use it as a base and will be rather quickly built using >>>>> those images as a cache. Once it is merged, I will add the automated build >>>>> in DockerHub and merge it to 1.10. As the next step I also plan to use the >>>>> prod image for our Kubernetes tests (that will help with moving to the >>>>> GitHub Actions from Travis). >>>>> >>>>> I added quite a comprehensive documentation - explaining all ins- and >>>>> outs- of both CI and PROD images >>>>> https://github.com/PolideaInternal/airflow/blob/add-production-docker-image/IMAGES.rst >>>>> . >>>>> >>>>> You can build it yourself using manual build process described in the >>>>> doc or (easier) using Breeze. That includes using the same Dockerfile to >>>>> build older Airflow version from the 1.10.* line. >>>>> >>>>> Feel free to use the new image in the Helm Chart you have - happy to >>>>> review the PRs. The image is available at >>>>> 'apache/airflow:master-python3.6' >>>>> and 'apache/airflow:master-python3.7` - once we merge it to 1.10 and >>>>> release 1.10.10 we will also have it automatically published for 1.10.10 >>>>> and beyond. >>>>> >>>>> Happy to hear your review comments and hope to merge it very soon. >>>>> J. >>>>> >>>>> On Thu, Mar 26, 2020 at 4:38 PM Jarek Potiuk <jarek.pot...@polidea.com> >>>>> wrote: >>>>> >>>>>> Absolutely! Please do :). The more eyes on this the better! >>>>>> >>>>>> On Thu, Mar 26, 2020 at 4:32 PM Daniel Imberman < >>>>>> daniel.imber...@gmail.com> wrote: >>>>>> >>>>>>> @Jarek Steven and I will look into getting KinD to work on GitHub >>>>>>> actions today. If we can get that working, we can donate the helm chart >>>>>>> and >>>>>>> possibly even deprecate the old Kubernetes tests in the process. >>>>>>> On Mar 26, 2020, 8:27 AM -0700, Jarek Potiuk < >>>>>>> jarek.pot...@polidea.com>, wrote: >>>>>>> > We have A POC for Github Actions done by Tomek, and the blocker >>>>>>> there >>>>>>> > was precisely Kubernetes tests and specifically running kind >>>>>>> cluster >>>>>>> > in the Github Actions >>>>>>> > https://github.com/PolideaInternal/airflow/pull/542 . In the >>>>>>> meantime >>>>>>> > Jiaje Zhong is trying to make it works on GA we discussed this very >>>>>>> > thing today at slack: >>>>>>> > >>>>>>> https://apache-airflow.slack.com/archives/CCPRP7943/p1585073249374200 >>>>>>> > >>>>>>> > For me this is the next thing I will look at after merging >>>>>>> > requirements PR and Prod image PR (and those two are a bit >>>>>>> > pre-requisites to make it stable and fast). I want to switch to the >>>>>>> > Production Image to run Kubernetes tests (they will be much faster) >>>>>>> > and then we can work out much faster how to run them on Github >>>>>>> > Actions. Then we can add Helm charts to run all those different >>>>>>> > deployments. I am really looking forward to it! >>>>>>> > >>>>>>> > I hope - this weekend I move forward all of that and have a PR >>>>>>> that we >>>>>>> > can start running tests on and replacing the resources with helm >>>>>>> > chart. >>>>>>> > >>>>>>> > J. >>>>>>> > >>>>>>> > On Thu, Mar 26, 2020 at 4:20 PM Daniel Imberman >>>>>>> > <daniel.imber...@gmail.com> wrote: >>>>>>> > > >>>>>>> > > @Steven glad to help you out on that. >>>>>>> > > >>>>>>> > > For the moment we only use travis, but I believe @jarek has been >>>>>>> looking to move to GitHub CI/CD. Perhaps we can use this as our first >>>>>>> GitHub tests? >>>>>>> > > On Mar 26, 2020, 8:17 AM -0700, Steven Miller >>>>>>> <ste...@astronomer.io.invalid>, wrote: >>>>>>> > > > Hey I’m happy to PR in the chart and get the CI working to >>>>>>> test it. Where >>>>>>> > > > do we want to run it? Do you all just use Travis only? >>>>>>> > > > >>>>>>> > > > Steven >>>>>>> > > > >>>>>>> > > > On Thu, Mar 26, 2020 at 10:59 AM Daniel Imberman < >>>>>>> daniel.imber...@gmail.com> >>>>>>> > > > wrote: >>>>>>> > > > >>>>>>> > > > > @Jarek I think with the helm chart + prod image we can go >>>>>>> even further >>>>>>> > > > > than that :). We can test CeleryExecutor, with KEDA >>>>>>> autoscaling, and a >>>>>>> > > > > bunch of other configurations. >>>>>>> > > > > On Mar 26, 2020, 7:45 AM -0700, Jarek Potiuk < >>>>>>> jarek.pot...@polidea.com>, >>>>>>> > > > > wrote: >>>>>>> > > > > > Yeah. I meant Custom Resources not CRDs in my original >>>>>>> email :) >>>>>>> > > > > > >>>>>>> > > > > > > On Thu, Mar 26, 2020 at 3:38 PM Daniel Imberman < >>>>>>> > > > > daniel.imber...@gmail.com> wrote: >>>>>>> > > > > > > > We’re not using CRDs for the tests at the moment. We >>>>>>> just have >>>>>>> > > > > deployment files. If anything having the helm chart as a >>>>>>> part of the >>>>>>> > > > > airflow repo could mean that the helm chart becomes the >>>>>>> defacto system for >>>>>>> > > > > testing airflow on kubernetes (we can get rid of all the >>>>>>> yams files and run >>>>>>> > > > > multiple k8s tests with different settings). >>>>>>> > > > > > > > On Mar 26, 2020, 7:20 AM -0700, Greg Neiheisel >>>>>>> > > > > <g...@astronomer.io.invalid>, wrote: >>>>>>> > > > > > > > > Yep, we can absolutely pull it into the airflow >>>>>>> repo. We've also >>>>>>> > > > > been >>>>>>> > > > > > > > > building up a test suite that currently runs on >>>>>>> CircleCI and uses >>>>>>> > > > > kind >>>>>>> > > > > > > > > (Kubernetes in Docker) to test several kubernetes >>>>>>> versions with >>>>>>> > > > > some >>>>>>> > > > > > > > > different settings. Right now we're mostly testing >>>>>>> the different >>>>>>> > > > > executors >>>>>>> > > > > > > > > since that has the biggest impact on what gets >>>>>>> deployed, but that >>>>>>> > > > > can be >>>>>>> > > > > > > > > expanded. >>>>>>> > > > > > > > > >>>>>>> > > > > > > > > What CRDs are currently being used to run Airflow >>>>>>> for the tests? >>>>>>> > > > > > > > > >>>>>>> > > > > > > > > On Wed, Mar 25, 2020 at 11:06 AM Jarek Potiuk < >>>>>>> > > > > jarek.pot...@polidea.com> >>>>>>> > > > > > > > > wrote: >>>>>>> > > > > > > > > >>>>>>> > > > > > > > > > One thing for the donation. >>>>>>> > > > > > > > > > >>>>>>> > > > > > > > > > Did you you want to have separate repository Greg ? >>>>>>> > > > > > > > > > >>>>>>> > > > > > > > > > I think we should simply create a folder in >>>>>>> Airflow repo and >>>>>>> > > > > keep it >>>>>>> > > > > > > > > > there (similarly as we keep Dockerfile). I am >>>>>>> going to switch our >>>>>>> > > > > > > > > > Kubernetes Tests to the production image (will >>>>>>> make the tests >>>>>>> > > > > much >>>>>>> > > > > > > > > > faster) and I am going to test the Dockerfile >>>>>>> automatically in >>>>>>> > > > > CI - >>>>>>> > > > > > > > > > for now we are using some custom Resource >>>>>>> definitions to start >>>>>>> > > > > Airflow >>>>>>> > > > > > > > > > on Kubernetes Cluster for the tests, but we could >>>>>>> switch to >>>>>>> > > > > using the >>>>>>> > > > > > > > > > helm chart - this way we can test all three things >>>>>>> at once: >>>>>>> > > > > > > > > > - Kubernetes Executor >>>>>>> > > > > > > > > > - Dockerfile >>>>>>> > > > > > > > > > - Helm Chart >>>>>>> > > > > > > > > > and we could also add more tests - for example >>>>>>> testing different >>>>>>> > > > > > > > > > deployment options for the helm chart. >>>>>>> > > > > > > > > > >>>>>>> > > > > > > > > > Having the Helm chart in Airflow repo would help >>>>>>> with that - >>>>>>> > > > > > > > > > especially in terms of future changes and testing >>>>>>> them >>>>>>> > > > > automatically. >>>>>>> > > > > > > > > > >>>>>>> > > > > > > > > > J. >>>>>>> > > > > > > > > > >>>>>>> > > > > > > > > > On Tue, Mar 24, 2020 at 9:09 PM Aizhamal Nurmamat >>>>>>> kyzy >>>>>>> > > > > > > > > > <aizha...@apache.org> wrote: >>>>>>> > > > > > > > > > > >>>>>>> > > > > > > > > > > +1 on the donation. Always happy to see more >>>>>>> useful stuff for >>>>>>> > > > > the >>>>>>> > > > > > > > > > > community :) >>>>>>> > > > > > > > > > > >>>>>>> > > > > > > > > > > On Tue, Mar 24, 2020 at 9:20 AM Greg Neiheisel < >>>>>>> > > > > g...@astronomer.io> >>>>>>> > > > > > > > > > wrote: >>>>>>> > > > > > > > > > > >>>>>>> > > > > > > > > > > > Yep, the cleanup_pods script is set up now as >>>>>>> an optional >>>>>>> > > > > Kubernetes >>>>>>> > > > > > > > > > > > CronJob ( >>>>>>> > > > > > > > > > > > >>>>>>> > > > > > > > > > >>>>>>> > > > > >>>>>>> https://github.com/astronomer/airflow-chart/blob/master/templates/cleanup/cleanup-cronjob.yaml >>>>>>> > > > > > > > > > ) >>>>>>> > > > > > > > > > > > that we have run periodically to clean failed >>>>>>> pods up and >>>>>>> > > > > could stay >>>>>>> > > > > > > > > > > > separate. >>>>>>> > > > > > > > > > > > >>>>>>> > > > > > > > > > > > The wait_for_migrations script could >>>>>>> definitely be pulled >>>>>>> > > > > into Airflow. >>>>>>> > > > > > > > > > > > For context, we deploy an initContainer on the >>>>>>> scheduler ( >>>>>>> > > > > > > > > > > > >>>>>>> > > > > > > > > > >>>>>>> > > > > >>>>>>> https://github.com/astronomer/airflow-chart/blob/master/templates/scheduler/scheduler-deployment.yaml#L77-L84 >>>>>>> > > > > > > > > > ) >>>>>>> > > > > > > > > > > > that runs the upgradedb command before booting >>>>>>> the >>>>>>> > > > > scheduler. This new >>>>>>> > > > > > > > > > > > wait_for_migration script runs in an >>>>>>> initContainer on the >>>>>>> > > > > webserver and >>>>>>> > > > > > > > > > > > workers ( >>>>>>> > > > > > > > > > > > >>>>>>> > > > > > > > > > >>>>>>> > > > > >>>>>>> https://github.com/astronomer/airflow-chart/blob/master/templates/webserver/webserver-deployment.yaml#L58-L65 >>>>>>> > > > > > > > > > ) >>>>>>> > > > > > > > > > > > so that they don't boot up ahead of a >>>>>>> potentially >>>>>>> > > > > long-running >>>>>>> > > > > > > > > > migration >>>>>>> > > > > > > > > > > > and attempt to operate on new or missing >>>>>>> columns/tables >>>>>>> > > > > before the >>>>>>> > > > > > > > > > > > migrations run. This prevents these pods from >>>>>>> entering a >>>>>>> > > > > CrashLoop. >>>>>>> > > > > > > > > > > > >>>>>>> > > > > > > > > > > > On Tue, Mar 24, 2020 at 11:48 AM Jarek Potiuk < >>>>>>> > > > > > > > > > jarek.pot...@polidea.com> >>>>>>> > > > > > > > > > > > wrote: >>>>>>> > > > > > > > > > > > >>>>>>> > > > > > > > > > > > > > >>>>>>> > > > > > > > > > > > > > @Tomasz great question. Our images are >>>>>>> currently >>>>>>> > > > > generated from >>>>>>> > > > > > > > > > > > > Dockerfiles >>>>>>> > > > > > > > > > > > > > in this repo >>>>>>> https://github.com/astronomer/ap-airflow >>>>>>> > > > > and get >>>>>>> > > > > > > > > > > > > published to >>>>>>> > > > > > > > > > > > > > DockerHub >>>>>>> > > > > > > > > > > > > > >>>>>>> > > > > >>>>>>> https://hub.docker.com/repository/docker/astronomerinc/ap-airflow. >>>>>>> > > > > > > > > > > > > > >>>>>>> > > > > > > > > > > > > > For the most part those are typical >>>>>>> Airflow images. >>>>>>> > > > > There's an >>>>>>> > > > > > > > > > > > > entrypoint >>>>>>> > > > > > > > > > > > > > script that we include in the image that >>>>>>> handles waiting >>>>>>> > > > > for the >>>>>>> > > > > > > > > > > > > database >>>>>>> > > > > > > > > > > > > > and redis (if used) to come up, which is >>>>>>> pretty generic. >>>>>>> > > > > > > > > > > > > >>>>>>> > > > > > > > > > > > > >>>>>>> > > > > > > > > > > > > I already added waiting for the database >>>>>>> (both metadata >>>>>>> > > > > and celery >>>>>>> > > > > > > > > > URL) in >>>>>>> > > > > > > > > > > > > the PR: >>>>>>> > > > > > > > > > > > > >>>>>>> > > > > > > > > > > > > >>>>>>> > > > > > > > > > >>>>>>> > > > > >>>>>>> https://github.com/apache/airflow/pull/7832/files#diff-3759f40d4e8ba0c0e82e82b66d376741 >>>>>>> > > > > > > > > > > > > . >>>>>>> > > > > > > > > > > > > It's functionally the same but more generic. >>>>>>> > > > > > > > > > > > > >>>>>>> > > > > > > > > > > > > The only other >>>>>>> > > > > > > > > > > > > > thing that I think the Helm Chart uses >>>>>>> would be the >>>>>>> > > > > scripts in this >>>>>>> > > > > > > > > > repo >>>>>>> > > > > > > > > > > > > > >>>>>>> https://github.com/astronomer/astronomer-airflow-scripts. >>>>>>> > > > > Our >>>>>>> > > > > > > > > > > > > Dockerfiles >>>>>>> > > > > > > > > > > > > > pull this package in. These scripts are >>>>>>> used to >>>>>>> > > > > coordinate running >>>>>>> > > > > > > > > > > > > > migrations and cleaning up failed pods. >>>>>>> > > > > > > > > > > > > > >>>>>>> > > > > > > > > > > > > >>>>>>> > > > > > > > > > > > > I see two scripts: >>>>>>> > > > > > > > > > > > > >>>>>>> > > > > > > > > > > > > * cleanup_pods -> this is (I believe) not >>>>>>> needed to run in >>>>>>> > > > > airflow - >>>>>>> > > > > > > > > > this >>>>>>> > > > > > > > > > > > > could be run as a separate pod/container? >>>>>>> > > > > > > > > > > > > * waiting for migrations -> I think this is >>>>>>> a good >>>>>>> > > > > candidate to add >>>>>>> > > > > > > > > > > > > *airflow >>>>>>> > > > > > > > > > > > > db wait_for_migration* command and make it >>>>>>> part of airflow >>>>>>> > > > > itself. >>>>>>> > > > > > > > > > > > > >>>>>>> > > > > > > > > > > > > I think we also have to agree on the Airflow >>>>>>> version >>>>>>> > > > > supported by the >>>>>>> > > > > > > > > > > > > official helm chart. I'd suggest we support >>>>>>> 1.10.10+ and we >>>>>>> > > > > > > > > > incorporate >>>>>>> > > > > > > > > > > > > all >>>>>>> > > > > > > > > > > > > the changes needed to airflow (like the "db >>>>>>> > > > > wait_for_migration") >>>>>>> > > > > > > > > > into 2.0 >>>>>>> > > > > > > > > > > > > and 1.10 and we support both - image and >>>>>>> helm chart for >>>>>>> > > > > those versions >>>>>>> > > > > > > > > > > > > only. That would help with people migrating >>>>>>> to the latest >>>>>>> > > > > version. >>>>>>> > > > > > > > > > > > > >>>>>>> > > > > > > > > > > > > WDYT? >>>>>>> > > > > > > > > > > > > >>>>>>> > > > > > > > > > > > > >>>>>>> > > > > > > > > > > > > > On Tue, Mar 24, 2020 at 10:49 AM Daniel >>>>>>> Imberman < >>>>>>> > > > > > > > > > > > > > daniel.imber...@gmail.com> >>>>>>> > > > > > > > > > > > > > wrote: >>>>>>> > > > > > > > > > > > > > >>>>>>> > > > > > > > > > > > > > > @jarek I agree completely. I think that >>>>>>> pairing an >>>>>>> > > > > official helm >>>>>>> > > > > > > > > > chart >>>>>>> > > > > > > > > > > > > > > with the official image would make for a >>>>>>> REALLY >>>>>>> > > > > powerful “up and >>>>>>> > > > > > > > > > > > > running >>>>>>> > > > > > > > > > > > > > > with airflow” story :). Tomek and I have >>>>>>> also been >>>>>>> > > > > looking into >>>>>>> > > > > > > > > > > > > > > operator-sdk which has the ability to >>>>>>> create custom >>>>>>> > > > > controllers >>>>>>> > > > > > > > > > from >>>>>>> > > > > > > > > > > > > helm >>>>>>> > > > > > > > > > > > > > > charts. We might even able to get a 1-2 >>>>>>> punch from the >>>>>>> > > > > same code >>>>>>> > > > > > > > > > base >>>>>>> > > > > > > > > > > > > :). >>>>>>> > > > > > > > > > > > > > > >>>>>>> > > > > > > > > > > > > > > @kaxil @jarek @aizhamal @ash if there’s >>>>>>> no issues, can >>>>>>> > > > > we please >>>>>>> > > > > > > > > > start >>>>>>> > > > > > > > > > > > > > the >>>>>>> > > > > > > > > > > > > > > process of donation? >>>>>>> > > > > > > > > > > > > > > >>>>>>> > > > > > > > > > > > > > > +1 on my part, of course :) >>>>>>> > > > > > > > > > > > > > > >>>>>>> > > > > > > > > > > > > > > >>>>>>> > > > > > > > > > > > > > > >>>>>>> > > > > > > > > > > > > > > Daniel >>>>>>> > > > > > > > > > > > > > > On Mar 24, 2020, 7:40 AM -0700, Jarek >>>>>>> Potiuk < >>>>>>> > > > > > > > > > > > > jarek.pot...@polidea.com>, >>>>>>> > > > > > > > > > > > > > > wrote: >>>>>>> > > > > > > > > > > > > > > > +1. And it should be paired with the >>>>>>> official image >>>>>>> > > > > we have >>>>>>> > > > > > > > > > work in >>>>>>> > > > > > > > > > > > > > > > progress on. I looked a lot at the >>>>>>> Astronomer's >>>>>>> > > > > image while >>>>>>> > > > > > > > > > > > > preparing >>>>>>> > > > > > > > > > > > > > my >>>>>>> > > > > > > > > > > > > > > > draft and we can make any adjustments >>>>>>> needed to make >>>>>>> > > > > it works >>>>>>> > > > > > > > > > with >>>>>>> > > > > > > > > > > > > the >>>>>>> > > > > > > > > > > > > > > helm >>>>>>> > > > > > > > > > > > > > > > chart - and I am super happy to >>>>>>> collaborate on that. >>>>>>> > > > > > > > > > > > > > > > >>>>>>> > > > > > > > > > > > > > > > PR here: >>>>>>> https://github.com/apache/airflow/pull/7832 >>>>>>> > > > > > > > > > > > > > > > >>>>>>> > > > > > > > > > > > > > > > J. >>>>>>> > > > > > > > > > > > > > > > >>>>>>> > > > > > > > > > > > > > > > >>>>>>> > > > > > > > > > > > > > > > On Tue, Mar 24, 2020 at 3:15 PM Kaxil >>>>>>> Naik < >>>>>>> > > > > kaxiln...@gmail.com >>>>>>> > > > > > > > > > > >>>>>>> > > > > > > > > > > > > > wrote: >>>>>>> > > > > > > > > > > > > > > > >>>>>>> > > > > > > > > > > > > > > > > @Tomasz Urbaszek < >>>>>>> tomasz.urbas...@polidea.com> : >>>>>>> > > > > > > > > > > > > > > > > Helm Chart Link: >>>>>>> > > > > https://github.com/astronomer/airflow-chart >>>>>>> > > > > > > > > > > > > > > > > >>>>>>> > > > > > > > > > > > > > > > > On Tue, Mar 24, 2020 at 2:13 PM >>>>>>> Tomasz Urbaszek < >>>>>>> > > > > > > > > > > > > > turbas...@apache.org> >>>>>>> > > > > > > > > > > > > > > > > wrote: >>>>>>> > > > > > > > > > > > > > > > > >>>>>>> > > > > > > > > > > > > > > > > > An official helm chart is >>>>>>> something our >>>>>>> > > > > community needs! >>>>>>> > > > > > > > > > Using >>>>>>> > > > > > > > > > > > > your >>>>>>> > > > > > > > > > > > > > > > > > chart as the official makes a lot >>>>>>> of sens to me >>>>>>> > > > > because as >>>>>>> > > > > > > > > > you >>>>>>> > > > > > > > > > > > > > > > > > mentioned - it's battle tested. >>>>>>> > > > > > > > > > > > > > > > > > >>>>>>> > > > > > > > > > > > > > > > > > One question: what Airflow image >>>>>>> do you use? >>>>>>> > > > > Also, would you >>>>>>> > > > > > > > > > > > > mind >>>>>>> > > > > > > > > > > > > > > > > > sharing a link to the chart? >>>>>>> > > > > > > > > > > > > > > > > > >>>>>>> > > > > > > > > > > > > > > > > > Tomek >>>>>>> > > > > > > > > > > > > > > > > > >>>>>>> > > > > > > > > > > > > > > > > > >>>>>>> > > > > > > > > > > > > > > > > > On Tue, Mar 24, 2020 at 2:07 PM >>>>>>> Greg Neiheisel >>>>>>> > > > > > > > > > > > > > > > > > <g...@astronomer.io.invalid> >>>>>>> wrote: >>>>>>> > > > > > > > > > > > > > > > > > > >>>>>>> > > > > > > > > > > > > > > > > > > Hey everyone, >>>>>>> > > > > > > > > > > > > > > > > > > >>>>>>> > > > > > > > > > > > > > > > > > > Over the past few years at >>>>>>> Astronomer, we’ve >>>>>>> > > > > created, >>>>>>> > > > > > > > > > managed, >>>>>>> > > > > > > > > > > > > > and >>>>>>> > > > > > > > > > > > > > > > > > hardened >>>>>>> > > > > > > > > > > > > > > > > > > a production-ready Helm Chart >>>>>>> for Airflow ( >>>>>>> > > > > > > > > > > > > > > > > > > >>>>>>> https://github.com/astronomer/airflow-chart) >>>>>>> > > > > that is >>>>>>> > > > > > > > > > being >>>>>>> > > > > > > > > > > > > used >>>>>>> > > > > > > > > > > > > > by >>>>>>> > > > > > > > > > > > > > > > > both >>>>>>> > > > > > > > > > > > > > > > > > our >>>>>>> > > > > > > > > > > > > > > > > > > SaaS and Enterprise customers. >>>>>>> This chart is >>>>>>> > > > > > > > > > battle-tested and >>>>>>> > > > > > > > > > > > > > > running >>>>>>> > > > > > > > > > > > > > > > > > > hundreds of Airflow deployments >>>>>>> of varying >>>>>>> > > > > sizes and >>>>>>> > > > > > > > > > runtime >>>>>>> > > > > > > > > > > > > > > > > > environments. >>>>>>> > > > > > > > > > > > > > > > > > > It’s been built up to >>>>>>> encapsulate the issues >>>>>>> > > > > that Airflow >>>>>>> > > > > > > > > > > > > users >>>>>>> > > > > > > > > > > > > > run >>>>>>> > > > > > > > > > > > > > > > > into >>>>>>> > > > > > > > > > > > > > > > > > in >>>>>>> > > > > > > > > > > > > > > > > > > the real world. >>>>>>> > > > > > > > > > > > > > > > > > > >>>>>>> > > > > > > > > > > > > > > > > > > While this chart was originally >>>>>>> developed >>>>>>> > > > > internally for >>>>>>> > > > > > > > > > our >>>>>>> > > > > > > > > > > > > > > Astronomer >>>>>>> > > > > > > > > > > > > > > > > > > Platform, we’ve recently >>>>>>> decoupled the chart >>>>>>> > > > > from the >>>>>>> > > > > > > > > > rest of >>>>>>> > > > > > > > > > > > > our >>>>>>> > > > > > > > > > > > > > > > > > platform >>>>>>> > > > > > > > > > > > > > > > > > > to make it usable by the greater >>>>>>> Airflow >>>>>>> > > > > community. With >>>>>>> > > > > > > > > > these >>>>>>> > > > > > > > > > > > > > > changes >>>>>>> > > > > > > > > > > > > > > > > in >>>>>>> > > > > > > > > > > > > > > > > > > mind, we want to start a >>>>>>> conversation about >>>>>>> > > > > donating this >>>>>>> > > > > > > > > > > > > chart >>>>>>> > > > > > > > > > > > > > to >>>>>>> > > > > > > > > > > > > > > the >>>>>>> > > > > > > > > > > > > > > > > > > Airflow community. >>>>>>> > > > > > > > > > > > > > > > > > > >>>>>>> > > > > > > > > > > > > > > > > > > Some of the main features of the >>>>>>> chart are: >>>>>>> > > > > > > > > > > > > > > > > > > >>>>>>> > > > > > > > > > > > > > > > > > > - It works out of the box. With >>>>>>> zero >>>>>>> > > > > configuration, a user >>>>>>> > > > > > > > > > > > > will >>>>>>> > > > > > > > > > > > > > get >>>>>>> > > > > > > > > > > > > > > > > a >>>>>>> > > > > > > > > > > > > > > > > > > postgres database, a default >>>>>>> user and the >>>>>>> > > > > > > > > > KubernetesExecutor >>>>>>> > > > > > > > > > > > > > ready >>>>>>> > > > > > > > > > > > > > > > > to >>>>>>> > > > > > > > > > > > > > > > > > run >>>>>>> > > > > > > > > > > > > > > > > > > DAGs. >>>>>>> > > > > > > > > > > > > > > > > > > - Support for Local, Celery (w/ >>>>>>> optional KEDA >>>>>>> > > > > > > > > > autoscaling) and >>>>>>> > > > > > > > > > > > > > > > > > > Kubernetes executors. >>>>>>> > > > > > > > > > > > > > > > > > > >>>>>>> > > > > > > > > > > > > > > > > > > Support for optional pgbouncer. >>>>>>> We use this to >>>>>>> > > > > share a >>>>>>> > > > > > > > > > > > > > configurable >>>>>>> > > > > > > > > > > > > > > > > > > connection pool size per >>>>>>> deployment. Useful >>>>>>> > > > > for limiting >>>>>>> > > > > > > > > > > > > > > connections to >>>>>>> > > > > > > > > > > > > > > > > > the >>>>>>> > > > > > > > > > > > > > > > > > > metadata database. >>>>>>> > > > > > > > > > > > > > > > > > > >>>>>>> > > > > > > > > > > > > > > > > > > - Airflow migration support. A >>>>>>> user can push a >>>>>>> > > > > newer >>>>>>> > > > > > > > > > version >>>>>>> > > > > > > > > > > > > of >>>>>>> > > > > > > > > > > > > > > > > > Airflow >>>>>>> > > > > > > > > > > > > > > > > > > into an existing release and >>>>>>> migrations will >>>>>>> > > > > > > > > > automatically run >>>>>>> > > > > > > > > > > > > > > > > > cleanly. >>>>>>> > > > > > > > > > > > > > > > > > > - Prometheus support. Optionally >>>>>>> install and >>>>>>> > > > > configure a >>>>>>> > > > > > > > > > > > > > > > > > statsd-exporter >>>>>>> > > > > > > > > > > > > > > > > > > to ingest Airflow metrics and >>>>>>> expose them to >>>>>>> > > > > Prometheus >>>>>>> > > > > > > > > > > > > > > > > automatically. >>>>>>> > > > > > > > > > > > > > > > > > > - Resource control. Optionally >>>>>>> control the >>>>>>> > > > > ResourceQuotas >>>>>>> > > > > > > > > > and >>>>>>> > > > > > > > > > > > > > > > > > > LimitRanges for each deployment >>>>>>> so that no >>>>>>> > > > > deployment can >>>>>>> > > > > > > > > > > > > > overload >>>>>>> > > > > > > > > > > > > > > a >>>>>>> > > > > > > > > > > > > > > > > > > cluster. >>>>>>> > > > > > > > > > > > > > > > > > > - Simple optional Elasticsearch >>>>>>> support. >>>>>>> > > > > > > > > > > > > > > > > > > - Optional namespace cleanup. >>>>>>> Sometimes >>>>>>> > > > > > > > > > KubernetesExecutor and >>>>>>> > > > > > > > > > > > > > > > > > > KubernetesPodOperator pods fail >>>>>>> for reasons >>>>>>> > > > > other than the >>>>>>> > > > > > > > > > > > > actual >>>>>>> > > > > > > > > > > > > > > > > > task. >>>>>>> > > > > > > > > > > > > > > > > > > This feature helps keep things >>>>>>> clean in >>>>>>> > > > > Kubernetes. >>>>>>> > > > > > > > > > > > > > > > > > > - Support for running locally in >>>>>>> KIND >>>>>>> > > > > (Kubernetes in >>>>>>> > > > > > > > > > Docker). >>>>>>> > > > > > > > > > > > > > > > > > > - Automatically tested across >>>>>>> many Kubernetes >>>>>>> > > > > versions >>>>>>> > > > > > > > > > with >>>>>>> > > > > > > > > > > > > Helm >>>>>>> > > > > > > > > > > > > > 2 >>>>>>> > > > > > > > > > > > > > > > > > and 3 >>>>>>> > > > > > > > > > > > > > > > > > > support. >>>>>>> > > > > > > > > > > > > > > > > > > >>>>>>> > > > > > > > > > > > > > > > > > > We’ve found that the cleanest >>>>>>> and most >>>>>>> > > > > reliable way to >>>>>>> > > > > > > > > > deploy >>>>>>> > > > > > > > > > > > > > DAGs >>>>>>> > > > > > > > > > > > > > > to >>>>>>> > > > > > > > > > > > > > > > > > > Kubernetes and manage them at >>>>>>> scale is to >>>>>>> > > > > package them >>>>>>> > > > > > > > > > into >>>>>>> > > > > > > > > > > > > the >>>>>>> > > > > > > > > > > > > > > actual >>>>>>> > > > > > > > > > > > > > > > > > > docker image, so we have geared >>>>>>> this chart >>>>>>> > > > > towards that >>>>>>> > > > > > > > > > > > > method of >>>>>>> > > > > > > > > > > > > > > > > > > operation, though adding other >>>>>>> methods should >>>>>>> > > > > be >>>>>>> > > > > > > > > > > > > straightforward. >>>>>>> > > > > > > > > > > > > > > > > > > >>>>>>> > > > > > > > > > > > > > > > > > > We would love thoughts from the >>>>>>> community and >>>>>>> > > > > would love >>>>>>> > > > > > > > > > to >>>>>>> > > > > > > > > > > > > see >>>>>>> > > > > > > > > > > > > > > this >>>>>>> > > > > > > > > > > > > > > > > > chart >>>>>>> > > > > > > > > > > > > > > > > > > help others to get up and >>>>>>> running on >>>>>>> > > > > Kubernetes! >>>>>>> > > > > > > > > > > > > > > > > > > >>>>>>> > > > > > > > > > > > > > > > > > > -- >>>>>>> > > > > > > > > > > > > > > > > > > *Greg Neiheisel* / Chief >>>>>>> Architect >>>>>>> > > > > Astronomer.io >>>>>>> > > > > > > > > > > > > > > > > > >>>>>>> > > > > > > > > > > > > > > > > >>>>>>> > > > > > > > > > > > > > > > >>>>>>> > > > > > > > > > > > > > > > >>>>>>> > > > > > > > > > > > > > > > -- >>>>>>> > > > > > > > > > > > > > > > >>>>>>> > > > > > > > > > > > > > > > Jarek Potiuk >>>>>>> > > > > > > > > > > > > > > > Polidea <https://www.polidea.com/> | >>>>>>> Principal >>>>>>> > > > > Software >>>>>>> > > > > > > > > > Engineer >>>>>>> > > > > > > > > > > > > > > > >>>>>>> > > > > > > > > > > > > > > > M: +48 660 796 129 >>>>>>> <+48%20660%20796%20129> >>>>>>> > > > > <+48660796129 >>>>>>> > > > > > > > > > > > > <+48%20660%20796%20129>> >>>>>>> > > > > > > > > > > > > > > > [image: Polidea] < >>>>>>> https://www.polidea.com/> >>>>>>> > > > > > > > > > > > > > > >>>>>>> > > > > > > > > > > > > > >>>>>>> > > > > > > > > > > > > > >>>>>>> > > > > > > > > > > > > > -- >>>>>>> > > > > > > > > > > > > > *Greg Neiheisel* / Chief Architect >>>>>>> Astronomer.io >>>>>>> > > > > > > > > > > > > > >>>>>>> > > > > > > > > > > > > >>>>>>> > > > > > > > > > > > > >>>>>>> > > > > > > > > > > > > -- >>>>>>> > > > > > > > > > > > > >>>>>>> > > > > > > > > > > > > Jarek Potiuk >>>>>>> > > > > > > > > > > > > Polidea <https://www.polidea.com/> | >>>>>>> Principal Software >>>>>>> > > > > Engineer >>>>>>> > > > > > > > > > > > > >>>>>>> > > > > > > > > > > > > M: +48 660 796 129 <+48%20660%20796%20129> >>>>>>> <+48660796129 >>>>>>> > > > > > > > > > > > > <+48%20660%20796%20129>> >>>>>>> > > > > > > > > > > > > [image: Polidea] <https://www.polidea.com/> >>>>>>> > > > > > > > > > > > > >>>>>>> > > > > > > > > > > > >>>>>>> > > > > > > > > > > > >>>>>>> > > > > > > > > > > > -- >>>>>>> > > > > > > > > > > > *Greg Neiheisel* / Chief Architect >>>>>>> Astronomer.io >>>>>>> > > > > > > > > > > > >>>>>>> > > > > > > > > > >>>>>>> > > > > > > > > > >>>>>>> > > > > > > > > > >>>>>>> > > > > > > > > > -- >>>>>>> > > > > > > > > > >>>>>>> > > > > > > > > > Jarek Potiuk >>>>>>> > > > > > > > > > Polidea | Principal Software Engineer >>>>>>> > > > > > > > > > >>>>>>> > > > > > > > > > M: +48 660 796 129 >>>>>>> > > > > > > > > > >>>>>>> > > > > > > > > >>>>>>> > > > > > > > > >>>>>>> > > > > > > > > -- >>>>>>> > > > > > > > > *Greg Neiheisel* / Chief Architect Astronomer.io >>>>>>> > > > > > >>>>>>> > > > > > >>>>>>> > > > > > -- >>>>>>> > > > > > Jarek Potiuk >>>>>>> > > > > > Polidea | Principal Software Engineer >>>>>>> > > > > > M: +48 660 796 129 >>>>>>> > > > > > >>>>>>> > > > > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > -- >>>>>>> > >>>>>>> > Jarek Potiuk >>>>>>> > Polidea | Principal Software Engineer >>>>>>> > >>>>>>> > M: +48 660 796 129 >>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> >>>>>> Jarek Potiuk >>>>>> Polidea <https://www.polidea.com/> | Principal Software Engineer >>>>>> >>>>>> M: +48 660 796 129 <+48660796129> >>>>>> [image: Polidea] <https://www.polidea.com/> >>>>>> >>>>>> >>>>> >>>>> -- >>>>> >>>>> Jarek Potiuk >>>>> Polidea <https://www.polidea.com/> | Principal Software Engineer >>>>> >>>>> M: +48 660 796 129 <+48660796129> >>>>> [image: Polidea] <https://www.polidea.com/> >>>>> >>>>> >>>> >>>> -- >>>> >>>> Jarek Potiuk >>>> Polidea <https://www.polidea.com/> | Principal Software Engineer >>>> >>>> M: +48 660 796 129 <+48660796129> >>>> [image: Polidea] <https://www.polidea.com/> >>>> >>>> >>> >>> -- >>> *Greg Neiheisel* / Chief Architect Astronomer.io >>> >> >> >> -- >> >> Jarek Potiuk >> Polidea <https://www.polidea.com/> | Principal Software Engineer >> >> M: +48 660 796 129 <+48660796129> >> [image: Polidea] <https://www.polidea.com/> >> >> > > -- > > Jarek Potiuk > Polidea <https://www.polidea.com/> | Principal Software Engineer > > M: +48 660 796 129 <+48660796129> > [image: Polidea] <https://www.polidea.com/> > > -- *Greg Neiheisel* / Chief Architect Astronomer.io
