Cool. Wil look into it over weekend. If you think anything can be improved in the image to make it a better fit - fill free to create new issue or comment the existing issues https://github.com/apache/airflow/issues?q=is%3Aopen+is%3Aissue+label%3Aarea%3Aproduction-image
On Fri, May 8, 2020 at 5:19 AM Greg Neiheisel <g...@astronomer.io> wrote: > Awesome Jarek. I just opened up a draft PR that adds the chart into the > Airflow repo. https://github.com/apache/airflow/pull/8777 > > It still contains some of our packaging, testing and releasing scripts. I > figure they can serve as a starting point to discuss how we'd want to > replicate into Airflow infrastructure. > > Looking forward to getting some feedback and moving this along! > > On Tue, Apr 28, 2020 at 8:00 AM Jarek Potiuk <jarek.pot...@polidea.com> > wrote: > >> Hey Greg, >> >> I just wanted to update you with some recent changes I've done to >> automate most of the image builds stuff we have in Airflow. That might be >> useful for your efforts. >> >> I am adding (and I will fix some of them very soon) issues based on >> feedback from people on the #prod-docker-image channel: >> https://github.com/apache/airflow/issues?q=is%3Aopen+is%3Aissue+label%3Aarea%3Aproduction-image >> But in the meantime all the build/CI infrastructure has nicely fallen into >> place and It's almost complete (one thing missing but very simple is >> automated builds of release images - for now this is manual but it is super >> easy change and I will add it before 1.10.11 for sure). >> >> I described the whole build infrastructure here: >> https://github.com/apache/airflow/blob/master/CI.rst including how image >> caching works, how nightly builds and images are triggered. I hope it will >> be helpful in the helm-chart work. Any questions or asks for the image - >> please let me know, it is super easy to update the image now when we have >> infrastructure to build it automatically integrated well with Github >> Actions and DockerHub. >> >> J. >> >> >> >> >> >> >> >> >> On Fri, Apr 10, 2020 at 7:23 PM Jarek Potiuk <jarek.pot...@polidea.com> >> wrote: >> >>> Awesome, thanks for the update! >>> >>> On Fri, Apr 10, 2020 at 6:57 PM Greg Neiheisel <g...@astronomer.io> >>> wrote: >>> >>>> Hey everyone, just wanted to post a quick update on the helm chart. >>>> We're in the process of removing any references to Astronomer specific >>>> things and making them more general features. That work is happening on >>>> this PR here https://github.com/astronomer/airflow-chart/pull/66. >>>> Hoping to wrap up the last few outstanding items and open up a PR to >>>> Airflow next week sometime! >>>> >>>> On Fri, Apr 3, 2020 at 10:19 AM Daniel Imberman < >>>> daniel.imber...@gmail.com> wrote: >>>> >>>>> Hi Jarek, >>>>> >>>>> Congrats on getting the image out! This is going to make a lot of >>>>> things easier :). We’ve actually been working on separating out KinD as >>>>> well (and in fact moving it out of travis and into GitHub issues). Let me >>>>> circle back with Steven (the engineer who’s been working on that) and get >>>>> a >>>>> status report from him. >>>>> >>>>> I will also take some time today to get our helm chart to work with >>>>> the airflow master image. It’s been a bit crazy around here in the last >>>>> few >>>>> weeks but I am consistently pushing to make sure we make the final changes >>>>> of ripping out any remaining platform components before we donate :). >>>>> >>>>> I agree that docker hub automation will be great. This will also allow >>>>> us to have a nightly build for any integration/UI tests we build. >>>>> >>>>> Thanks again Jarek! >>>>> >>>>> Daniel >>>>> >>>>> via Newton Mail >>>>> <https://cloudmagic.com/k/d/mailapp?ct=dx&cv=10.0.32&pv=10.14.6&source=email_footer_2> >>>>> >>>>> On Fri, Apr 3, 2020 at 1:04 AM, Jarek Potiuk <jarek.pot...@polidea.com> >>>>> wrote: >>>>> >>>>> Hello Greg and Daniel, >>>>> >>>>> Production image is merged and it's also going to be released in >>>>> 1.10.10. For now, I will manually tag the release in DockerHub when we >>>>> release but we will automate that in the future. Currently, we have >>>>> *airflow:master* and *airflow:v1-10-test* images (for all supported >>>>> python versions) that you can use for testing with the Helm chart! >>>>> >>>>> The next things from my side: >>>>> >>>>> * I will automate DockerHub building for the prod image from master - >>>>> for now only CI is automated but we need to add prod image as well >>>>> * I will move kind-cluster creation outside of the container and use >>>>> production image for our CI Kubernetes tests. I have a WIP branch on it >>>>> and >>>>> I think I should have it early next week >>>>> * The next step will be to switch our CI to Github Actions finally. I >>>>> think we will run both Travis and GA for a short time to make sure it >>>>> works >>>>> * I will ask people for options they would like to have in the image - >>>>> additional packages etc. so that I can work on the next iteration of the >>>>> image. Also if you have any needs to see for the Helm integration, I am >>>>> super happy to incorporate them in the next version. I plan to keep the >>>>> "master" and "v1-10-test" images updated with any new changes and push >>>>> images regularly >>>>> * I am happy to help with the Helm Chart, so let me know if you need >>>>> anything! >>>>> >>>>> J. >>>>> >>>>> >>>>> On Mon, Mar 30, 2020 at 11:21 PM Jarek Potiuk < >>>>> jarek.pot...@polidea.com> wrote: >>>>> >>>>>> Hello Daniel and Greg. >>>>>> >>>>>> I think the production image for Airflow is quite ready for the final >>>>>> review and merge. I did some tests and I think it's quite complete now. >>>>>> >>>>>> The PR is running it's Travis build: >>>>>> https://github.com/apache/airflow/pull/7832 >>>>>> >>>>>> I already pushed the image to DockerHub - so that the next time it is >>>>>> merged It will use it as a base and will be rather quickly built using >>>>>> those images as a cache. Once it is merged, I will add the automated >>>>>> build >>>>>> in DockerHub and merge it to 1.10. As the next step I also plan to use >>>>>> the >>>>>> prod image for our Kubernetes tests (that will help with moving to the >>>>>> GitHub Actions from Travis). >>>>>> >>>>>> I added quite a comprehensive documentation - explaining all ins- and >>>>>> outs- of both CI and PROD images >>>>>> https://github.com/PolideaInternal/airflow/blob/add-production-docker-image/IMAGES.rst >>>>>> . >>>>>> >>>>>> You can build it yourself using manual build process described in the >>>>>> doc or (easier) using Breeze. That includes using the same Dockerfile to >>>>>> build older Airflow version from the 1.10.* line. >>>>>> >>>>>> Feel free to use the new image in the Helm Chart you have - happy to >>>>>> review the PRs. The image is available at >>>>>> 'apache/airflow:master-python3.6' >>>>>> and 'apache/airflow:master-python3.7` - once we merge it to 1.10 and >>>>>> release 1.10.10 we will also have it automatically published for 1.10.10 >>>>>> and beyond. >>>>>> >>>>>> Happy to hear your review comments and hope to merge it very soon. >>>>>> J. >>>>>> >>>>>> On Thu, Mar 26, 2020 at 4:38 PM Jarek Potiuk < >>>>>> jarek.pot...@polidea.com> wrote: >>>>>> >>>>>>> Absolutely! Please do :). The more eyes on this the better! >>>>>>> >>>>>>> On Thu, Mar 26, 2020 at 4:32 PM Daniel Imberman < >>>>>>> daniel.imber...@gmail.com> wrote: >>>>>>> >>>>>>>> @Jarek Steven and I will look into getting KinD to work on GitHub >>>>>>>> actions today. If we can get that working, we can donate the helm >>>>>>>> chart and >>>>>>>> possibly even deprecate the old Kubernetes tests in the process. >>>>>>>> On Mar 26, 2020, 8:27 AM -0700, Jarek Potiuk < >>>>>>>> jarek.pot...@polidea.com>, wrote: >>>>>>>> > We have A POC for Github Actions done by Tomek, and the blocker >>>>>>>> there >>>>>>>> > was precisely Kubernetes tests and specifically running kind >>>>>>>> cluster >>>>>>>> > in the Github Actions >>>>>>>> > https://github.com/PolideaInternal/airflow/pull/542 . In the >>>>>>>> meantime >>>>>>>> > Jiaje Zhong is trying to make it works on GA we discussed this >>>>>>>> very >>>>>>>> > thing today at slack: >>>>>>>> > >>>>>>>> https://apache-airflow.slack.com/archives/CCPRP7943/p1585073249374200 >>>>>>>> > >>>>>>>> > For me this is the next thing I will look at after merging >>>>>>>> > requirements PR and Prod image PR (and those two are a bit >>>>>>>> > pre-requisites to make it stable and fast). I want to switch to >>>>>>>> the >>>>>>>> > Production Image to run Kubernetes tests (they will be much >>>>>>>> faster) >>>>>>>> > and then we can work out much faster how to run them on Github >>>>>>>> > Actions. Then we can add Helm charts to run all those different >>>>>>>> > deployments. I am really looking forward to it! >>>>>>>> > >>>>>>>> > I hope - this weekend I move forward all of that and have a PR >>>>>>>> that we >>>>>>>> > can start running tests on and replacing the resources with helm >>>>>>>> > chart. >>>>>>>> > >>>>>>>> > J. >>>>>>>> > >>>>>>>> > On Thu, Mar 26, 2020 at 4:20 PM Daniel Imberman >>>>>>>> > <daniel.imber...@gmail.com> wrote: >>>>>>>> > > >>>>>>>> > > @Steven glad to help you out on that. >>>>>>>> > > >>>>>>>> > > For the moment we only use travis, but I believe @jarek has >>>>>>>> been looking to move to GitHub CI/CD. Perhaps we can use this as our >>>>>>>> first >>>>>>>> GitHub tests? >>>>>>>> > > On Mar 26, 2020, 8:17 AM -0700, Steven Miller >>>>>>>> <ste...@astronomer.io.invalid>, wrote: >>>>>>>> > > > Hey I’m happy to PR in the chart and get the CI working to >>>>>>>> test it. Where >>>>>>>> > > > do we want to run it? Do you all just use Travis only? >>>>>>>> > > > >>>>>>>> > > > Steven >>>>>>>> > > > >>>>>>>> > > > On Thu, Mar 26, 2020 at 10:59 AM Daniel Imberman < >>>>>>>> daniel.imber...@gmail.com> >>>>>>>> > > > wrote: >>>>>>>> > > > >>>>>>>> > > > > @Jarek I think with the helm chart + prod image we can go >>>>>>>> even further >>>>>>>> > > > > than that :). We can test CeleryExecutor, with KEDA >>>>>>>> autoscaling, and a >>>>>>>> > > > > bunch of other configurations. >>>>>>>> > > > > On Mar 26, 2020, 7:45 AM -0700, Jarek Potiuk < >>>>>>>> jarek.pot...@polidea.com>, >>>>>>>> > > > > wrote: >>>>>>>> > > > > > Yeah. I meant Custom Resources not CRDs in my original >>>>>>>> email :) >>>>>>>> > > > > > >>>>>>>> > > > > > > On Thu, Mar 26, 2020 at 3:38 PM Daniel Imberman < >>>>>>>> > > > > daniel.imber...@gmail.com> wrote: >>>>>>>> > > > > > > > We’re not using CRDs for the tests at the moment. We >>>>>>>> just have >>>>>>>> > > > > deployment files. If anything having the helm chart as a >>>>>>>> part of the >>>>>>>> > > > > airflow repo could mean that the helm chart becomes the >>>>>>>> defacto system for >>>>>>>> > > > > testing airflow on kubernetes (we can get rid of all the >>>>>>>> yams files and run >>>>>>>> > > > > multiple k8s tests with different settings). >>>>>>>> > > > > > > > On Mar 26, 2020, 7:20 AM -0700, Greg Neiheisel >>>>>>>> > > > > <g...@astronomer.io.invalid>, wrote: >>>>>>>> > > > > > > > > Yep, we can absolutely pull it into the airflow >>>>>>>> repo. We've also >>>>>>>> > > > > been >>>>>>>> > > > > > > > > building up a test suite that currently runs on >>>>>>>> CircleCI and uses >>>>>>>> > > > > kind >>>>>>>> > > > > > > > > (Kubernetes in Docker) to test several kubernetes >>>>>>>> versions with >>>>>>>> > > > > some >>>>>>>> > > > > > > > > different settings. Right now we're mostly testing >>>>>>>> the different >>>>>>>> > > > > executors >>>>>>>> > > > > > > > > since that has the biggest impact on what gets >>>>>>>> deployed, but that >>>>>>>> > > > > can be >>>>>>>> > > > > > > > > expanded. >>>>>>>> > > > > > > > > >>>>>>>> > > > > > > > > What CRDs are currently being used to run Airflow >>>>>>>> for the tests? >>>>>>>> > > > > > > > > >>>>>>>> > > > > > > > > On Wed, Mar 25, 2020 at 11:06 AM Jarek Potiuk < >>>>>>>> > > > > jarek.pot...@polidea.com> >>>>>>>> > > > > > > > > wrote: >>>>>>>> > > > > > > > > >>>>>>>> > > > > > > > > > One thing for the donation. >>>>>>>> > > > > > > > > > >>>>>>>> > > > > > > > > > Did you you want to have separate repository Greg >>>>>>>> ? >>>>>>>> > > > > > > > > > >>>>>>>> > > > > > > > > > I think we should simply create a folder in >>>>>>>> Airflow repo and >>>>>>>> > > > > keep it >>>>>>>> > > > > > > > > > there (similarly as we keep Dockerfile). I am >>>>>>>> going to switch our >>>>>>>> > > > > > > > > > Kubernetes Tests to the production image (will >>>>>>>> make the tests >>>>>>>> > > > > much >>>>>>>> > > > > > > > > > faster) and I am going to test the Dockerfile >>>>>>>> automatically in >>>>>>>> > > > > CI - >>>>>>>> > > > > > > > > > for now we are using some custom Resource >>>>>>>> definitions to start >>>>>>>> > > > > Airflow >>>>>>>> > > > > > > > > > on Kubernetes Cluster for the tests, but we could >>>>>>>> switch to >>>>>>>> > > > > using the >>>>>>>> > > > > > > > > > helm chart - this way we can test all three >>>>>>>> things at once: >>>>>>>> > > > > > > > > > - Kubernetes Executor >>>>>>>> > > > > > > > > > - Dockerfile >>>>>>>> > > > > > > > > > - Helm Chart >>>>>>>> > > > > > > > > > and we could also add more tests - for example >>>>>>>> testing different >>>>>>>> > > > > > > > > > deployment options for the helm chart. >>>>>>>> > > > > > > > > > >>>>>>>> > > > > > > > > > Having the Helm chart in Airflow repo would help >>>>>>>> with that - >>>>>>>> > > > > > > > > > especially in terms of future changes and testing >>>>>>>> them >>>>>>>> > > > > automatically. >>>>>>>> > > > > > > > > > >>>>>>>> > > > > > > > > > J. >>>>>>>> > > > > > > > > > >>>>>>>> > > > > > > > > > On Tue, Mar 24, 2020 at 9:09 PM Aizhamal Nurmamat >>>>>>>> kyzy >>>>>>>> > > > > > > > > > <aizha...@apache.org> wrote: >>>>>>>> > > > > > > > > > > >>>>>>>> > > > > > > > > > > +1 on the donation. Always happy to see more >>>>>>>> useful stuff for >>>>>>>> > > > > the >>>>>>>> > > > > > > > > > > community :) >>>>>>>> > > > > > > > > > > >>>>>>>> > > > > > > > > > > On Tue, Mar 24, 2020 at 9:20 AM Greg Neiheisel < >>>>>>>> > > > > g...@astronomer.io> >>>>>>>> > > > > > > > > > wrote: >>>>>>>> > > > > > > > > > > >>>>>>>> > > > > > > > > > > > Yep, the cleanup_pods script is set up now as >>>>>>>> an optional >>>>>>>> > > > > Kubernetes >>>>>>>> > > > > > > > > > > > CronJob ( >>>>>>>> > > > > > > > > > > > >>>>>>>> > > > > > > > > > >>>>>>>> > > > > >>>>>>>> https://github.com/astronomer/airflow-chart/blob/master/templates/cleanup/cleanup-cronjob.yaml >>>>>>>> > > > > > > > > > ) >>>>>>>> > > > > > > > > > > > that we have run periodically to clean failed >>>>>>>> pods up and >>>>>>>> > > > > could stay >>>>>>>> > > > > > > > > > > > separate. >>>>>>>> > > > > > > > > > > > >>>>>>>> > > > > > > > > > > > The wait_for_migrations script could >>>>>>>> definitely be pulled >>>>>>>> > > > > into Airflow. >>>>>>>> > > > > > > > > > > > For context, we deploy an initContainer on >>>>>>>> the scheduler ( >>>>>>>> > > > > > > > > > > > >>>>>>>> > > > > > > > > > >>>>>>>> > > > > >>>>>>>> https://github.com/astronomer/airflow-chart/blob/master/templates/scheduler/scheduler-deployment.yaml#L77-L84 >>>>>>>> > > > > > > > > > ) >>>>>>>> > > > > > > > > > > > that runs the upgradedb command before >>>>>>>> booting the >>>>>>>> > > > > scheduler. This new >>>>>>>> > > > > > > > > > > > wait_for_migration script runs in an >>>>>>>> initContainer on the >>>>>>>> > > > > webserver and >>>>>>>> > > > > > > > > > > > workers ( >>>>>>>> > > > > > > > > > > > >>>>>>>> > > > > > > > > > >>>>>>>> > > > > >>>>>>>> https://github.com/astronomer/airflow-chart/blob/master/templates/webserver/webserver-deployment.yaml#L58-L65 >>>>>>>> > > > > > > > > > ) >>>>>>>> > > > > > > > > > > > so that they don't boot up ahead of a >>>>>>>> potentially >>>>>>>> > > > > long-running >>>>>>>> > > > > > > > > > migration >>>>>>>> > > > > > > > > > > > and attempt to operate on new or missing >>>>>>>> columns/tables >>>>>>>> > > > > before the >>>>>>>> > > > > > > > > > > > migrations run. This prevents these pods from >>>>>>>> entering a >>>>>>>> > > > > CrashLoop. >>>>>>>> > > > > > > > > > > > >>>>>>>> > > > > > > > > > > > On Tue, Mar 24, 2020 at 11:48 AM Jarek Potiuk >>>>>>>> < >>>>>>>> > > > > > > > > > jarek.pot...@polidea.com> >>>>>>>> > > > > > > > > > > > wrote: >>>>>>>> > > > > > > > > > > > >>>>>>>> > > > > > > > > > > > > > >>>>>>>> > > > > > > > > > > > > > @Tomasz great question. Our images are >>>>>>>> currently >>>>>>>> > > > > generated from >>>>>>>> > > > > > > > > > > > > Dockerfiles >>>>>>>> > > > > > > > > > > > > > in this repo >>>>>>>> https://github.com/astronomer/ap-airflow >>>>>>>> > > > > and get >>>>>>>> > > > > > > > > > > > > published to >>>>>>>> > > > > > > > > > > > > > DockerHub >>>>>>>> > > > > > > > > > > > > > >>>>>>>> > > > > >>>>>>>> https://hub.docker.com/repository/docker/astronomerinc/ap-airflow. >>>>>>>> > > > > > > > > > > > > > >>>>>>>> > > > > > > > > > > > > > For the most part those are typical >>>>>>>> Airflow images. >>>>>>>> > > > > There's an >>>>>>>> > > > > > > > > > > > > entrypoint >>>>>>>> > > > > > > > > > > > > > script that we include in the image that >>>>>>>> handles waiting >>>>>>>> > > > > for the >>>>>>>> > > > > > > > > > > > > database >>>>>>>> > > > > > > > > > > > > > and redis (if used) to come up, which is >>>>>>>> pretty generic. >>>>>>>> > > > > > > > > > > > > >>>>>>>> > > > > > > > > > > > > >>>>>>>> > > > > > > > > > > > > I already added waiting for the database >>>>>>>> (both metadata >>>>>>>> > > > > and celery >>>>>>>> > > > > > > > > > URL) in >>>>>>>> > > > > > > > > > > > > the PR: >>>>>>>> > > > > > > > > > > > > >>>>>>>> > > > > > > > > > > > > >>>>>>>> > > > > > > > > > >>>>>>>> > > > > >>>>>>>> https://github.com/apache/airflow/pull/7832/files#diff-3759f40d4e8ba0c0e82e82b66d376741 >>>>>>>> > > > > > > > > > > > > . >>>>>>>> > > > > > > > > > > > > It's functionally the same but more generic. >>>>>>>> > > > > > > > > > > > > >>>>>>>> > > > > > > > > > > > > The only other >>>>>>>> > > > > > > > > > > > > > thing that I think the Helm Chart uses >>>>>>>> would be the >>>>>>>> > > > > scripts in this >>>>>>>> > > > > > > > > > repo >>>>>>>> > > > > > > > > > > > > > >>>>>>>> https://github.com/astronomer/astronomer-airflow-scripts. >>>>>>>> > > > > Our >>>>>>>> > > > > > > > > > > > > Dockerfiles >>>>>>>> > > > > > > > > > > > > > pull this package in. These scripts are >>>>>>>> used to >>>>>>>> > > > > coordinate running >>>>>>>> > > > > > > > > > > > > > migrations and cleaning up failed pods. >>>>>>>> > > > > > > > > > > > > > >>>>>>>> > > > > > > > > > > > > >>>>>>>> > > > > > > > > > > > > I see two scripts: >>>>>>>> > > > > > > > > > > > > >>>>>>>> > > > > > > > > > > > > * cleanup_pods -> this is (I believe) not >>>>>>>> needed to run in >>>>>>>> > > > > airflow - >>>>>>>> > > > > > > > > > this >>>>>>>> > > > > > > > > > > > > could be run as a separate pod/container? >>>>>>>> > > > > > > > > > > > > * waiting for migrations -> I think this is >>>>>>>> a good >>>>>>>> > > > > candidate to add >>>>>>>> > > > > > > > > > > > > *airflow >>>>>>>> > > > > > > > > > > > > db wait_for_migration* command and make it >>>>>>>> part of airflow >>>>>>>> > > > > itself. >>>>>>>> > > > > > > > > > > > > >>>>>>>> > > > > > > > > > > > > I think we also have to agree on the >>>>>>>> Airflow version >>>>>>>> > > > > supported by the >>>>>>>> > > > > > > > > > > > > official helm chart. I'd suggest we support >>>>>>>> 1.10.10+ and we >>>>>>>> > > > > > > > > > incorporate >>>>>>>> > > > > > > > > > > > > all >>>>>>>> > > > > > > > > > > > > the changes needed to airflow (like the "db >>>>>>>> > > > > wait_for_migration") >>>>>>>> > > > > > > > > > into 2.0 >>>>>>>> > > > > > > > > > > > > and 1.10 and we support both - image and >>>>>>>> helm chart for >>>>>>>> > > > > those versions >>>>>>>> > > > > > > > > > > > > only. That would help with people migrating >>>>>>>> to the latest >>>>>>>> > > > > version. >>>>>>>> > > > > > > > > > > > > >>>>>>>> > > > > > > > > > > > > WDYT? >>>>>>>> > > > > > > > > > > > > >>>>>>>> > > > > > > > > > > > > >>>>>>>> > > > > > > > > > > > > > On Tue, Mar 24, 2020 at 10:49 AM Daniel >>>>>>>> Imberman < >>>>>>>> > > > > > > > > > > > > > daniel.imber...@gmail.com> >>>>>>>> > > > > > > > > > > > > > wrote: >>>>>>>> > > > > > > > > > > > > > >>>>>>>> > > > > > > > > > > > > > > @jarek I agree completely. I think that >>>>>>>> pairing an >>>>>>>> > > > > official helm >>>>>>>> > > > > > > > > > chart >>>>>>>> > > > > > > > > > > > > > > with the official image would make for >>>>>>>> a REALLY >>>>>>>> > > > > powerful “up and >>>>>>>> > > > > > > > > > > > > running >>>>>>>> > > > > > > > > > > > > > > with airflow” story :). Tomek and I >>>>>>>> have also been >>>>>>>> > > > > looking into >>>>>>>> > > > > > > > > > > > > > > operator-sdk which has the ability to >>>>>>>> create custom >>>>>>>> > > > > controllers >>>>>>>> > > > > > > > > > from >>>>>>>> > > > > > > > > > > > > helm >>>>>>>> > > > > > > > > > > > > > > charts. We might even able to get a 1-2 >>>>>>>> punch from the >>>>>>>> > > > > same code >>>>>>>> > > > > > > > > > base >>>>>>>> > > > > > > > > > > > > :). >>>>>>>> > > > > > > > > > > > > > > >>>>>>>> > > > > > > > > > > > > > > @kaxil @jarek @aizhamal @ash if there’s >>>>>>>> no issues, can >>>>>>>> > > > > we please >>>>>>>> > > > > > > > > > start >>>>>>>> > > > > > > > > > > > > > the >>>>>>>> > > > > > > > > > > > > > > process of donation? >>>>>>>> > > > > > > > > > > > > > > >>>>>>>> > > > > > > > > > > > > > > +1 on my part, of course :) >>>>>>>> > > > > > > > > > > > > > > >>>>>>>> > > > > > > > > > > > > > > >>>>>>>> > > > > > > > > > > > > > > >>>>>>>> > > > > > > > > > > > > > > Daniel >>>>>>>> > > > > > > > > > > > > > > On Mar 24, 2020, 7:40 AM -0700, Jarek >>>>>>>> Potiuk < >>>>>>>> > > > > > > > > > > > > jarek.pot...@polidea.com>, >>>>>>>> > > > > > > > > > > > > > > wrote: >>>>>>>> > > > > > > > > > > > > > > > +1. And it should be paired with the >>>>>>>> official image >>>>>>>> > > > > we have >>>>>>>> > > > > > > > > > work in >>>>>>>> > > > > > > > > > > > > > > > progress on. I looked a lot at the >>>>>>>> Astronomer's >>>>>>>> > > > > image while >>>>>>>> > > > > > > > > > > > > preparing >>>>>>>> > > > > > > > > > > > > > my >>>>>>>> > > > > > > > > > > > > > > > draft and we can make any adjustments >>>>>>>> needed to make >>>>>>>> > > > > it works >>>>>>>> > > > > > > > > > with >>>>>>>> > > > > > > > > > > > > the >>>>>>>> > > > > > > > > > > > > > > helm >>>>>>>> > > > > > > > > > > > > > > > chart - and I am super happy to >>>>>>>> collaborate on that. >>>>>>>> > > > > > > > > > > > > > > > >>>>>>>> > > > > > > > > > > > > > > > PR here: >>>>>>>> https://github.com/apache/airflow/pull/7832 >>>>>>>> > > > > > > > > > > > > > > > >>>>>>>> > > > > > > > > > > > > > > > J. >>>>>>>> > > > > > > > > > > > > > > > >>>>>>>> > > > > > > > > > > > > > > > >>>>>>>> > > > > > > > > > > > > > > > On Tue, Mar 24, 2020 at 3:15 PM Kaxil >>>>>>>> Naik < >>>>>>>> > > > > kaxiln...@gmail.com >>>>>>>> > > > > > > > > > > >>>>>>>> > > > > > > > > > > > > > wrote: >>>>>>>> > > > > > > > > > > > > > > > >>>>>>>> > > > > > > > > > > > > > > > > @Tomasz Urbaszek < >>>>>>>> tomasz.urbas...@polidea.com> : >>>>>>>> > > > > > > > > > > > > > > > > Helm Chart Link: >>>>>>>> > > > > https://github.com/astronomer/airflow-chart >>>>>>>> > > > > > > > > > > > > > > > > >>>>>>>> > > > > > > > > > > > > > > > > On Tue, Mar 24, 2020 at 2:13 PM >>>>>>>> Tomasz Urbaszek < >>>>>>>> > > > > > > > > > > > > > turbas...@apache.org> >>>>>>>> > > > > > > > > > > > > > > > > wrote: >>>>>>>> > > > > > > > > > > > > > > > > >>>>>>>> > > > > > > > > > > > > > > > > > An official helm chart is >>>>>>>> something our >>>>>>>> > > > > community needs! >>>>>>>> > > > > > > > > > Using >>>>>>>> > > > > > > > > > > > > your >>>>>>>> > > > > > > > > > > > > > > > > > chart as the official makes a lot >>>>>>>> of sens to me >>>>>>>> > > > > because as >>>>>>>> > > > > > > > > > you >>>>>>>> > > > > > > > > > > > > > > > > > mentioned - it's battle tested. >>>>>>>> > > > > > > > > > > > > > > > > > >>>>>>>> > > > > > > > > > > > > > > > > > One question: what Airflow image >>>>>>>> do you use? >>>>>>>> > > > > Also, would you >>>>>>>> > > > > > > > > > > > > mind >>>>>>>> > > > > > > > > > > > > > > > > > sharing a link to the chart? >>>>>>>> > > > > > > > > > > > > > > > > > >>>>>>>> > > > > > > > > > > > > > > > > > Tomek >>>>>>>> > > > > > > > > > > > > > > > > > >>>>>>>> > > > > > > > > > > > > > > > > > >>>>>>>> > > > > > > > > > > > > > > > > > On Tue, Mar 24, 2020 at 2:07 PM >>>>>>>> Greg Neiheisel >>>>>>>> > > > > > > > > > > > > > > > > > <g...@astronomer.io.invalid> >>>>>>>> wrote: >>>>>>>> > > > > > > > > > > > > > > > > > > >>>>>>>> > > > > > > > > > > > > > > > > > > Hey everyone, >>>>>>>> > > > > > > > > > > > > > > > > > > >>>>>>>> > > > > > > > > > > > > > > > > > > Over the past few years at >>>>>>>> Astronomer, we’ve >>>>>>>> > > > > created, >>>>>>>> > > > > > > > > > managed, >>>>>>>> > > > > > > > > > > > > > and >>>>>>>> > > > > > > > > > > > > > > > > > hardened >>>>>>>> > > > > > > > > > > > > > > > > > > a production-ready Helm Chart >>>>>>>> for Airflow ( >>>>>>>> > > > > > > > > > > > > > > > > > > >>>>>>>> https://github.com/astronomer/airflow-chart) >>>>>>>> > > > > that is >>>>>>>> > > > > > > > > > being >>>>>>>> > > > > > > > > > > > > used >>>>>>>> > > > > > > > > > > > > > by >>>>>>>> > > > > > > > > > > > > > > > > both >>>>>>>> > > > > > > > > > > > > > > > > > our >>>>>>>> > > > > > > > > > > > > > > > > > > SaaS and Enterprise customers. >>>>>>>> This chart is >>>>>>>> > > > > > > > > > battle-tested and >>>>>>>> > > > > > > > > > > > > > > running >>>>>>>> > > > > > > > > > > > > > > > > > > hundreds of Airflow deployments >>>>>>>> of varying >>>>>>>> > > > > sizes and >>>>>>>> > > > > > > > > > runtime >>>>>>>> > > > > > > > > > > > > > > > > > environments. >>>>>>>> > > > > > > > > > > > > > > > > > > It’s been built up to >>>>>>>> encapsulate the issues >>>>>>>> > > > > that Airflow >>>>>>>> > > > > > > > > > > > > users >>>>>>>> > > > > > > > > > > > > > run >>>>>>>> > > > > > > > > > > > > > > > > into >>>>>>>> > > > > > > > > > > > > > > > > > in >>>>>>>> > > > > > > > > > > > > > > > > > > the real world. >>>>>>>> > > > > > > > > > > > > > > > > > > >>>>>>>> > > > > > > > > > > > > > > > > > > While this chart was originally >>>>>>>> developed >>>>>>>> > > > > internally for >>>>>>>> > > > > > > > > > our >>>>>>>> > > > > > > > > > > > > > > Astronomer >>>>>>>> > > > > > > > > > > > > > > > > > > Platform, we’ve recently >>>>>>>> decoupled the chart >>>>>>>> > > > > from the >>>>>>>> > > > > > > > > > rest of >>>>>>>> > > > > > > > > > > > > our >>>>>>>> > > > > > > > > > > > > > > > > > platform >>>>>>>> > > > > > > > > > > > > > > > > > > to make it usable by the >>>>>>>> greater Airflow >>>>>>>> > > > > community. With >>>>>>>> > > > > > > > > > these >>>>>>>> > > > > > > > > > > > > > > changes >>>>>>>> > > > > > > > > > > > > > > > > in >>>>>>>> > > > > > > > > > > > > > > > > > > mind, we want to start a >>>>>>>> conversation about >>>>>>>> > > > > donating this >>>>>>>> > > > > > > > > > > > > chart >>>>>>>> > > > > > > > > > > > > > to >>>>>>>> > > > > > > > > > > > > > > the >>>>>>>> > > > > > > > > > > > > > > > > > > Airflow community. >>>>>>>> > > > > > > > > > > > > > > > > > > >>>>>>>> > > > > > > > > > > > > > > > > > > Some of the main features of >>>>>>>> the chart are: >>>>>>>> > > > > > > > > > > > > > > > > > > >>>>>>>> > > > > > > > > > > > > > > > > > > - It works out of the box. With >>>>>>>> zero >>>>>>>> > > > > configuration, a user >>>>>>>> > > > > > > > > > > > > will >>>>>>>> > > > > > > > > > > > > > get >>>>>>>> > > > > > > > > > > > > > > > > a >>>>>>>> > > > > > > > > > > > > > > > > > > postgres database, a default >>>>>>>> user and the >>>>>>>> > > > > > > > > > KubernetesExecutor >>>>>>>> > > > > > > > > > > > > > ready >>>>>>>> > > > > > > > > > > > > > > > > to >>>>>>>> > > > > > > > > > > > > > > > > > run >>>>>>>> > > > > > > > > > > > > > > > > > > DAGs. >>>>>>>> > > > > > > > > > > > > > > > > > > - Support for Local, Celery (w/ >>>>>>>> optional KEDA >>>>>>>> > > > > > > > > > autoscaling) and >>>>>>>> > > > > > > > > > > > > > > > > > > Kubernetes executors. >>>>>>>> > > > > > > > > > > > > > > > > > > >>>>>>>> > > > > > > > > > > > > > > > > > > Support for optional pgbouncer. >>>>>>>> We use this to >>>>>>>> > > > > share a >>>>>>>> > > > > > > > > > > > > > configurable >>>>>>>> > > > > > > > > > > > > > > > > > > connection pool size per >>>>>>>> deployment. Useful >>>>>>>> > > > > for limiting >>>>>>>> > > > > > > > > > > > > > > connections to >>>>>>>> > > > > > > > > > > > > > > > > > the >>>>>>>> > > > > > > > > > > > > > > > > > > metadata database. >>>>>>>> > > > > > > > > > > > > > > > > > > >>>>>>>> > > > > > > > > > > > > > > > > > > - Airflow migration support. A >>>>>>>> user can push a >>>>>>>> > > > > newer >>>>>>>> > > > > > > > > > version >>>>>>>> > > > > > > > > > > > > of >>>>>>>> > > > > > > > > > > > > > > > > > Airflow >>>>>>>> > > > > > > > > > > > > > > > > > > into an existing release and >>>>>>>> migrations will >>>>>>>> > > > > > > > > > automatically run >>>>>>>> > > > > > > > > > > > > > > > > > cleanly. >>>>>>>> > > > > > > > > > > > > > > > > > > - Prometheus support. >>>>>>>> Optionally install and >>>>>>>> > > > > configure a >>>>>>>> > > > > > > > > > > > > > > > > > statsd-exporter >>>>>>>> > > > > > > > > > > > > > > > > > > to ingest Airflow metrics and >>>>>>>> expose them to >>>>>>>> > > > > Prometheus >>>>>>>> > > > > > > > > > > > > > > > > automatically. >>>>>>>> > > > > > > > > > > > > > > > > > > - Resource control. Optionally >>>>>>>> control the >>>>>>>> > > > > ResourceQuotas >>>>>>>> > > > > > > > > > and >>>>>>>> > > > > > > > > > > > > > > > > > > LimitRanges for each deployment >>>>>>>> so that no >>>>>>>> > > > > deployment can >>>>>>>> > > > > > > > > > > > > > overload >>>>>>>> > > > > > > > > > > > > > > a >>>>>>>> > > > > > > > > > > > > > > > > > > cluster. >>>>>>>> > > > > > > > > > > > > > > > > > > - Simple optional Elasticsearch >>>>>>>> support. >>>>>>>> > > > > > > > > > > > > > > > > > > - Optional namespace cleanup. >>>>>>>> Sometimes >>>>>>>> > > > > > > > > > KubernetesExecutor and >>>>>>>> > > > > > > > > > > > > > > > > > > KubernetesPodOperator pods fail >>>>>>>> for reasons >>>>>>>> > > > > other than the >>>>>>>> > > > > > > > > > > > > actual >>>>>>>> > > > > > > > > > > > > > > > > > task. >>>>>>>> > > > > > > > > > > > > > > > > > > This feature helps keep things >>>>>>>> clean in >>>>>>>> > > > > Kubernetes. >>>>>>>> > > > > > > > > > > > > > > > > > > - Support for running locally >>>>>>>> in KIND >>>>>>>> > > > > (Kubernetes in >>>>>>>> > > > > > > > > > Docker). >>>>>>>> > > > > > > > > > > > > > > > > > > - Automatically tested across >>>>>>>> many Kubernetes >>>>>>>> > > > > versions >>>>>>>> > > > > > > > > > with >>>>>>>> > > > > > > > > > > > > Helm >>>>>>>> > > > > > > > > > > > > > 2 >>>>>>>> > > > > > > > > > > > > > > > > > and 3 >>>>>>>> > > > > > > > > > > > > > > > > > > support. >>>>>>>> > > > > > > > > > > > > > > > > > > >>>>>>>> > > > > > > > > > > > > > > > > > > We’ve found that the cleanest >>>>>>>> and most >>>>>>>> > > > > reliable way to >>>>>>>> > > > > > > > > > deploy >>>>>>>> > > > > > > > > > > > > > DAGs >>>>>>>> > > > > > > > > > > > > > > to >>>>>>>> > > > > > > > > > > > > > > > > > > Kubernetes and manage them at >>>>>>>> scale is to >>>>>>>> > > > > package them >>>>>>>> > > > > > > > > > into >>>>>>>> > > > > > > > > > > > > the >>>>>>>> > > > > > > > > > > > > > > actual >>>>>>>> > > > > > > > > > > > > > > > > > > docker image, so we have geared >>>>>>>> this chart >>>>>>>> > > > > towards that >>>>>>>> > > > > > > > > > > > > method of >>>>>>>> > > > > > > > > > > > > > > > > > > operation, though adding other >>>>>>>> methods should >>>>>>>> > > > > be >>>>>>>> > > > > > > > > > > > > straightforward. >>>>>>>> > > > > > > > > > > > > > > > > > > >>>>>>>> > > > > > > > > > > > > > > > > > > We would love thoughts from the >>>>>>>> community and >>>>>>>> > > > > would love >>>>>>>> > > > > > > > > > to >>>>>>>> > > > > > > > > > > > > see >>>>>>>> > > > > > > > > > > > > > > this >>>>>>>> > > > > > > > > > > > > > > > > > chart >>>>>>>> > > > > > > > > > > > > > > > > > > help others to get up and >>>>>>>> running on >>>>>>>> > > > > Kubernetes! >>>>>>>> > > > > > > > > > > > > > > > > > > >>>>>>>> > > > > > > > > > > > > > > > > > > -- >>>>>>>> > > > > > > > > > > > > > > > > > > *Greg Neiheisel* / Chief >>>>>>>> Architect >>>>>>>> > > > > Astronomer.io >>>>>>>> > > > > > > > > > > > > > > > > > >>>>>>>> > > > > > > > > > > > > > > > > >>>>>>>> > > > > > > > > > > > > > > > >>>>>>>> > > > > > > > > > > > > > > > >>>>>>>> > > > > > > > > > > > > > > > -- >>>>>>>> > > > > > > > > > > > > > > > >>>>>>>> > > > > > > > > > > > > > > > Jarek Potiuk >>>>>>>> > > > > > > > > > > > > > > > Polidea <https://www.polidea.com/> | >>>>>>>> Principal >>>>>>>> > > > > Software >>>>>>>> > > > > > > > > > Engineer >>>>>>>> > > > > > > > > > > > > > > > >>>>>>>> > > > > > > > > > > > > > > > M: +48 660 796 129 >>>>>>>> <+48%20660%20796%20129> >>>>>>>> > > > > <+48660796129 >>>>>>>> > > > > > > > > > > > > <+48%20660%20796%20129>> >>>>>>>> > > > > > > > > > > > > > > > [image: Polidea] < >>>>>>>> https://www.polidea.com/> >>>>>>>> > > > > > > > > > > > > > > >>>>>>>> > > > > > > > > > > > > > >>>>>>>> > > > > > > > > > > > > > >>>>>>>> > > > > > > > > > > > > > -- >>>>>>>> > > > > > > > > > > > > > *Greg Neiheisel* / Chief Architect >>>>>>>> Astronomer.io >>>>>>>> > > > > > > > > > > > > > >>>>>>>> > > > > > > > > > > > > >>>>>>>> > > > > > > > > > > > > >>>>>>>> > > > > > > > > > > > > -- >>>>>>>> > > > > > > > > > > > > >>>>>>>> > > > > > > > > > > > > Jarek Potiuk >>>>>>>> > > > > > > > > > > > > Polidea <https://www.polidea.com/> | >>>>>>>> Principal Software >>>>>>>> > > > > Engineer >>>>>>>> > > > > > > > > > > > > >>>>>>>> > > > > > > > > > > > > M: +48 660 796 129 <+48%20660%20796%20129> >>>>>>>> <+48660796129 >>>>>>>> > > > > > > > > > > > > <+48%20660%20796%20129>> >>>>>>>> > > > > > > > > > > > > [image: Polidea] <https://www.polidea.com/> >>>>>>>> > > > > > > > > > > > > >>>>>>>> > > > > > > > > > > > >>>>>>>> > > > > > > > > > > > >>>>>>>> > > > > > > > > > > > -- >>>>>>>> > > > > > > > > > > > *Greg Neiheisel* / Chief Architect >>>>>>>> Astronomer.io >>>>>>>> > > > > > > > > > > > >>>>>>>> > > > > > > > > > >>>>>>>> > > > > > > > > > >>>>>>>> > > > > > > > > > >>>>>>>> > > > > > > > > > -- >>>>>>>> > > > > > > > > > >>>>>>>> > > > > > > > > > Jarek Potiuk >>>>>>>> > > > > > > > > > Polidea | Principal Software Engineer >>>>>>>> > > > > > > > > > >>>>>>>> > > > > > > > > > M: +48 660 796 129 >>>>>>>> > > > > > > > > > >>>>>>>> > > > > > > > > >>>>>>>> > > > > > > > > >>>>>>>> > > > > > > > > -- >>>>>>>> > > > > > > > > *Greg Neiheisel* / Chief Architect Astronomer.io >>>>>>>> > > > > > >>>>>>>> > > > > > >>>>>>>> > > > > > -- >>>>>>>> > > > > > Jarek Potiuk >>>>>>>> > > > > > Polidea | Principal Software Engineer >>>>>>>> > > > > > M: +48 660 796 129 >>>>>>>> > > > > > >>>>>>>> > > > > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > -- >>>>>>>> > >>>>>>>> > Jarek Potiuk >>>>>>>> > Polidea | Principal Software Engineer >>>>>>>> > >>>>>>>> > M: +48 660 796 129 >>>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> >>>>>>> Jarek Potiuk >>>>>>> Polidea <https://www.polidea.com/> | Principal Software Engineer >>>>>>> >>>>>>> M: +48 660 796 129 <+48660796129> >>>>>>> [image: Polidea] <https://www.polidea.com/> >>>>>>> >>>>>>> >>>>>> >>>>>> -- >>>>>> >>>>>> Jarek Potiuk >>>>>> Polidea <https://www.polidea.com/> | Principal Software Engineer >>>>>> >>>>>> M: +48 660 796 129 <+48660796129> >>>>>> [image: Polidea] <https://www.polidea.com/> >>>>>> >>>>>> >>>>> >>>>> -- >>>>> >>>>> Jarek Potiuk >>>>> Polidea <https://www.polidea.com/> | Principal Software Engineer >>>>> >>>>> M: +48 660 796 129 <+48660796129> >>>>> [image: Polidea] <https://www.polidea.com/> >>>>> >>>>> >>>> >>>> -- >>>> *Greg Neiheisel* / Chief Architect Astronomer.io >>>> >>> >>> >>> -- >>> >>> Jarek Potiuk >>> Polidea <https://www.polidea.com/> | Principal Software Engineer >>> >>> M: +48 660 796 129 <+48660796129> >>> [image: Polidea] <https://www.polidea.com/> >>> >>> >> >> -- >> >> Jarek Potiuk >> Polidea <https://www.polidea.com/> | Principal Software Engineer >> >> M: +48 660 796 129 <+48660796129> >> [image: Polidea] <https://www.polidea.com/> >> >> > > -- > *Greg Neiheisel* / Chief Architect Astronomer.io > -- Jarek Potiuk Polidea <https://www.polidea.com/> | Principal Software Engineer M: +48 660 796 129 <+48660796129> [image: Polidea] <https://www.polidea.com/>
