Hello Greg and Daniel, Production image is merged and it's also going to be released in 1.10.10. For now, I will manually tag the release in DockerHub when we release but we will automate that in the future. Currently, we have *airflow:master* and *airflow:v1-10-test* images (for all supported python versions) that you can use for testing with the Helm chart!
The next things from my side: * I will automate DockerHub building for the prod image from master - for now only CI is automated but we need to add prod image as well * I will move kind-cluster creation outside of the container and use production image for our CI Kubernetes tests. I have a WIP branch on it and I think I should have it early next week * The next step will be to switch our CI to Github Actions finally. I think we will run both Travis and GA for a short time to make sure it works * I will ask people for options they would like to have in the image - additional packages etc. so that I can work on the next iteration of the image. Also if you have any needs to see for the Helm integration, I am super happy to incorporate them in the next version. I plan to keep the "master" and "v1-10-test" images updated with any new changes and push images regularly * I am happy to help with the Helm Chart, so let me know if you need anything! J. On Mon, Mar 30, 2020 at 11:21 PM Jarek Potiuk <jarek.pot...@polidea.com> wrote: > Hello Daniel and Greg. > > I think the production image for Airflow is quite ready for the final > review and merge. I did some tests and I think it's quite complete now. > > The PR is running it's Travis build: > https://github.com/apache/airflow/pull/7832 > > I already pushed the image to DockerHub - so that the next time it is > merged It will use it as a base and will be rather quickly built using > those images as a cache. Once it is merged, I will add the automated build > in DockerHub and merge it to 1.10. As the next step I also plan to use the > prod image for our Kubernetes tests (that will help with moving to the > GitHub Actions from Travis). > > I added quite a comprehensive documentation - explaining all ins- and > outs- of both CI and PROD images > https://github.com/PolideaInternal/airflow/blob/add-production-docker-image/IMAGES.rst > . > > You can build it yourself using manual build process described in the doc > or (easier) using Breeze. That includes using the same Dockerfile to > build older Airflow version from the 1.10.* line. > > Feel free to use the new image in the Helm Chart you have - happy to > review the PRs. The image is available at 'apache/airflow:master-python3.6' > and 'apache/airflow:master-python3.7` - once we merge it to 1.10 and > release 1.10.10 we will also have it automatically published for 1.10.10 > and beyond. > > Happy to hear your review comments and hope to merge it very soon. > J. > > On Thu, Mar 26, 2020 at 4:38 PM Jarek Potiuk <jarek.pot...@polidea.com> > wrote: > >> Absolutely! Please do :). The more eyes on this the better! >> >> On Thu, Mar 26, 2020 at 4:32 PM Daniel Imberman < >> daniel.imber...@gmail.com> wrote: >> >>> @Jarek Steven and I will look into getting KinD to work on GitHub >>> actions today. If we can get that working, we can donate the helm chart and >>> possibly even deprecate the old Kubernetes tests in the process. >>> On Mar 26, 2020, 8:27 AM -0700, Jarek Potiuk <jarek.pot...@polidea.com>, >>> wrote: >>> > We have A POC for Github Actions done by Tomek, and the blocker there >>> > was precisely Kubernetes tests and specifically running kind cluster >>> > in the Github Actions >>> > https://github.com/PolideaInternal/airflow/pull/542 . In the meantime >>> > Jiaje Zhong is trying to make it works on GA we discussed this very >>> > thing today at slack: >>> > https://apache-airflow.slack.com/archives/CCPRP7943/p1585073249374200 >>> > >>> > For me this is the next thing I will look at after merging >>> > requirements PR and Prod image PR (and those two are a bit >>> > pre-requisites to make it stable and fast). I want to switch to the >>> > Production Image to run Kubernetes tests (they will be much faster) >>> > and then we can work out much faster how to run them on Github >>> > Actions. Then we can add Helm charts to run all those different >>> > deployments. I am really looking forward to it! >>> > >>> > I hope - this weekend I move forward all of that and have a PR that we >>> > can start running tests on and replacing the resources with helm >>> > chart. >>> > >>> > J. >>> > >>> > On Thu, Mar 26, 2020 at 4:20 PM Daniel Imberman >>> > <daniel.imber...@gmail.com> wrote: >>> > > >>> > > @Steven glad to help you out on that. >>> > > >>> > > For the moment we only use travis, but I believe @jarek has been >>> looking to move to GitHub CI/CD. Perhaps we can use this as our first >>> GitHub tests? >>> > > On Mar 26, 2020, 8:17 AM -0700, Steven Miller >>> <ste...@astronomer.io.invalid>, wrote: >>> > > > Hey I’m happy to PR in the chart and get the CI working to test >>> it. Where >>> > > > do we want to run it? Do you all just use Travis only? >>> > > > >>> > > > Steven >>> > > > >>> > > > On Thu, Mar 26, 2020 at 10:59 AM Daniel Imberman < >>> daniel.imber...@gmail.com> >>> > > > wrote: >>> > > > >>> > > > > @Jarek I think with the helm chart + prod image we can go even >>> further >>> > > > > than that :). We can test CeleryExecutor, with KEDA autoscaling, >>> and a >>> > > > > bunch of other configurations. >>> > > > > On Mar 26, 2020, 7:45 AM -0700, Jarek Potiuk < >>> jarek.pot...@polidea.com>, >>> > > > > wrote: >>> > > > > > Yeah. I meant Custom Resources not CRDs in my original email :) >>> > > > > > >>> > > > > > > On Thu, Mar 26, 2020 at 3:38 PM Daniel Imberman < >>> > > > > daniel.imber...@gmail.com> wrote: >>> > > > > > > > We’re not using CRDs for the tests at the moment. We just >>> have >>> > > > > deployment files. If anything having the helm chart as a part of >>> the >>> > > > > airflow repo could mean that the helm chart becomes the defacto >>> system for >>> > > > > testing airflow on kubernetes (we can get rid of all the yams >>> files and run >>> > > > > multiple k8s tests with different settings). >>> > > > > > > > On Mar 26, 2020, 7:20 AM -0700, Greg Neiheisel >>> > > > > <g...@astronomer.io.invalid>, wrote: >>> > > > > > > > > Yep, we can absolutely pull it into the airflow repo. >>> We've also >>> > > > > been >>> > > > > > > > > building up a test suite that currently runs on CircleCI >>> and uses >>> > > > > kind >>> > > > > > > > > (Kubernetes in Docker) to test several kubernetes >>> versions with >>> > > > > some >>> > > > > > > > > different settings. Right now we're mostly testing the >>> different >>> > > > > executors >>> > > > > > > > > since that has the biggest impact on what gets deployed, >>> but that >>> > > > > can be >>> > > > > > > > > expanded. >>> > > > > > > > > >>> > > > > > > > > What CRDs are currently being used to run Airflow for >>> the tests? >>> > > > > > > > > >>> > > > > > > > > On Wed, Mar 25, 2020 at 11:06 AM Jarek Potiuk < >>> > > > > jarek.pot...@polidea.com> >>> > > > > > > > > wrote: >>> > > > > > > > > >>> > > > > > > > > > One thing for the donation. >>> > > > > > > > > > >>> > > > > > > > > > Did you you want to have separate repository Greg ? >>> > > > > > > > > > >>> > > > > > > > > > I think we should simply create a folder in Airflow >>> repo and >>> > > > > keep it >>> > > > > > > > > > there (similarly as we keep Dockerfile). I am going to >>> switch our >>> > > > > > > > > > Kubernetes Tests to the production image (will make >>> the tests >>> > > > > much >>> > > > > > > > > > faster) and I am going to test the Dockerfile >>> automatically in >>> > > > > CI - >>> > > > > > > > > > for now we are using some custom Resource definitions >>> to start >>> > > > > Airflow >>> > > > > > > > > > on Kubernetes Cluster for the tests, but we could >>> switch to >>> > > > > using the >>> > > > > > > > > > helm chart - this way we can test all three things at >>> once: >>> > > > > > > > > > - Kubernetes Executor >>> > > > > > > > > > - Dockerfile >>> > > > > > > > > > - Helm Chart >>> > > > > > > > > > and we could also add more tests - for example testing >>> different >>> > > > > > > > > > deployment options for the helm chart. >>> > > > > > > > > > >>> > > > > > > > > > Having the Helm chart in Airflow repo would help with >>> that - >>> > > > > > > > > > especially in terms of future changes and testing them >>> > > > > automatically. >>> > > > > > > > > > >>> > > > > > > > > > J. >>> > > > > > > > > > >>> > > > > > > > > > On Tue, Mar 24, 2020 at 9:09 PM Aizhamal Nurmamat kyzy >>> > > > > > > > > > <aizha...@apache.org> wrote: >>> > > > > > > > > > > >>> > > > > > > > > > > +1 on the donation. Always happy to see more useful >>> stuff for >>> > > > > the >>> > > > > > > > > > > community :) >>> > > > > > > > > > > >>> > > > > > > > > > > On Tue, Mar 24, 2020 at 9:20 AM Greg Neiheisel < >>> > > > > g...@astronomer.io> >>> > > > > > > > > > wrote: >>> > > > > > > > > > > >>> > > > > > > > > > > > Yep, the cleanup_pods script is set up now as an >>> optional >>> > > > > Kubernetes >>> > > > > > > > > > > > CronJob ( >>> > > > > > > > > > > > >>> > > > > > > > > > >>> > > > > >>> https://github.com/astronomer/airflow-chart/blob/master/templates/cleanup/cleanup-cronjob.yaml >>> > > > > > > > > > ) >>> > > > > > > > > > > > that we have run periodically to clean failed pods >>> up and >>> > > > > could stay >>> > > > > > > > > > > > separate. >>> > > > > > > > > > > > >>> > > > > > > > > > > > The wait_for_migrations script could definitely be >>> pulled >>> > > > > into Airflow. >>> > > > > > > > > > > > For context, we deploy an initContainer on the >>> scheduler ( >>> > > > > > > > > > > > >>> > > > > > > > > > >>> > > > > >>> https://github.com/astronomer/airflow-chart/blob/master/templates/scheduler/scheduler-deployment.yaml#L77-L84 >>> > > > > > > > > > ) >>> > > > > > > > > > > > that runs the upgradedb command before booting the >>> > > > > scheduler. This new >>> > > > > > > > > > > > wait_for_migration script runs in an initContainer >>> on the >>> > > > > webserver and >>> > > > > > > > > > > > workers ( >>> > > > > > > > > > > > >>> > > > > > > > > > >>> > > > > >>> https://github.com/astronomer/airflow-chart/blob/master/templates/webserver/webserver-deployment.yaml#L58-L65 >>> > > > > > > > > > ) >>> > > > > > > > > > > > so that they don't boot up ahead of a potentially >>> > > > > long-running >>> > > > > > > > > > migration >>> > > > > > > > > > > > and attempt to operate on new or missing >>> columns/tables >>> > > > > before the >>> > > > > > > > > > > > migrations run. This prevents these pods from >>> entering a >>> > > > > CrashLoop. >>> > > > > > > > > > > > >>> > > > > > > > > > > > On Tue, Mar 24, 2020 at 11:48 AM Jarek Potiuk < >>> > > > > > > > > > jarek.pot...@polidea.com> >>> > > > > > > > > > > > wrote: >>> > > > > > > > > > > > >>> > > > > > > > > > > > > > >>> > > > > > > > > > > > > > @Tomasz great question. Our images are >>> currently >>> > > > > generated from >>> > > > > > > > > > > > > Dockerfiles >>> > > > > > > > > > > > > > in this repo >>> https://github.com/astronomer/ap-airflow >>> > > > > and get >>> > > > > > > > > > > > > published to >>> > > > > > > > > > > > > > DockerHub >>> > > > > > > > > > > > > > >>> > > > > >>> https://hub.docker.com/repository/docker/astronomerinc/ap-airflow. >>> > > > > > > > > > > > > > >>> > > > > > > > > > > > > > For the most part those are typical Airflow >>> images. >>> > > > > There's an >>> > > > > > > > > > > > > entrypoint >>> > > > > > > > > > > > > > script that we include in the image that >>> handles waiting >>> > > > > for the >>> > > > > > > > > > > > > database >>> > > > > > > > > > > > > > and redis (if used) to come up, which is >>> pretty generic. >>> > > > > > > > > > > > > >>> > > > > > > > > > > > > >>> > > > > > > > > > > > > I already added waiting for the database (both >>> metadata >>> > > > > and celery >>> > > > > > > > > > URL) in >>> > > > > > > > > > > > > the PR: >>> > > > > > > > > > > > > >>> > > > > > > > > > > > > >>> > > > > > > > > > >>> > > > > >>> https://github.com/apache/airflow/pull/7832/files#diff-3759f40d4e8ba0c0e82e82b66d376741 >>> > > > > > > > > > > > > . >>> > > > > > > > > > > > > It's functionally the same but more generic. >>> > > > > > > > > > > > > >>> > > > > > > > > > > > > The only other >>> > > > > > > > > > > > > > thing that I think the Helm Chart uses would >>> be the >>> > > > > scripts in this >>> > > > > > > > > > repo >>> > > > > > > > > > > > > > >>> https://github.com/astronomer/astronomer-airflow-scripts. >>> > > > > Our >>> > > > > > > > > > > > > Dockerfiles >>> > > > > > > > > > > > > > pull this package in. These scripts are used to >>> > > > > coordinate running >>> > > > > > > > > > > > > > migrations and cleaning up failed pods. >>> > > > > > > > > > > > > > >>> > > > > > > > > > > > > >>> > > > > > > > > > > > > I see two scripts: >>> > > > > > > > > > > > > >>> > > > > > > > > > > > > * cleanup_pods -> this is (I believe) not needed >>> to run in >>> > > > > airflow - >>> > > > > > > > > > this >>> > > > > > > > > > > > > could be run as a separate pod/container? >>> > > > > > > > > > > > > * waiting for migrations -> I think this is a >>> good >>> > > > > candidate to add >>> > > > > > > > > > > > > *airflow >>> > > > > > > > > > > > > db wait_for_migration* command and make it part >>> of airflow >>> > > > > itself. >>> > > > > > > > > > > > > >>> > > > > > > > > > > > > I think we also have to agree on the Airflow >>> version >>> > > > > supported by the >>> > > > > > > > > > > > > official helm chart. I'd suggest we support >>> 1.10.10+ and we >>> > > > > > > > > > incorporate >>> > > > > > > > > > > > > all >>> > > > > > > > > > > > > the changes needed to airflow (like the "db >>> > > > > wait_for_migration") >>> > > > > > > > > > into 2.0 >>> > > > > > > > > > > > > and 1.10 and we support both - image and helm >>> chart for >>> > > > > those versions >>> > > > > > > > > > > > > only. That would help with people migrating to >>> the latest >>> > > > > version. >>> > > > > > > > > > > > > >>> > > > > > > > > > > > > WDYT? >>> > > > > > > > > > > > > >>> > > > > > > > > > > > > >>> > > > > > > > > > > > > > On Tue, Mar 24, 2020 at 10:49 AM Daniel >>> Imberman < >>> > > > > > > > > > > > > > daniel.imber...@gmail.com> >>> > > > > > > > > > > > > > wrote: >>> > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > @jarek I agree completely. I think that >>> pairing an >>> > > > > official helm >>> > > > > > > > > > chart >>> > > > > > > > > > > > > > > with the official image would make for a >>> REALLY >>> > > > > powerful “up and >>> > > > > > > > > > > > > running >>> > > > > > > > > > > > > > > with airflow” story :). Tomek and I have >>> also been >>> > > > > looking into >>> > > > > > > > > > > > > > > operator-sdk which has the ability to create >>> custom >>> > > > > controllers >>> > > > > > > > > > from >>> > > > > > > > > > > > > helm >>> > > > > > > > > > > > > > > charts. We might even able to get a 1-2 >>> punch from the >>> > > > > same code >>> > > > > > > > > > base >>> > > > > > > > > > > > > :). >>> > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > @kaxil @jarek @aizhamal @ash if there’s no >>> issues, can >>> > > > > we please >>> > > > > > > > > > start >>> > > > > > > > > > > > > > the >>> > > > > > > > > > > > > > > process of donation? >>> > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > +1 on my part, of course :) >>> > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > Daniel >>> > > > > > > > > > > > > > > On Mar 24, 2020, 7:40 AM -0700, Jarek Potiuk >>> < >>> > > > > > > > > > > > > jarek.pot...@polidea.com>, >>> > > > > > > > > > > > > > > wrote: >>> > > > > > > > > > > > > > > > +1. And it should be paired with the >>> official image >>> > > > > we have >>> > > > > > > > > > work in >>> > > > > > > > > > > > > > > > progress on. I looked a lot at the >>> Astronomer's >>> > > > > image while >>> > > > > > > > > > > > > preparing >>> > > > > > > > > > > > > > my >>> > > > > > > > > > > > > > > > draft and we can make any adjustments >>> needed to make >>> > > > > it works >>> > > > > > > > > > with >>> > > > > > > > > > > > > the >>> > > > > > > > > > > > > > > helm >>> > > > > > > > > > > > > > > > chart - and I am super happy to >>> collaborate on that. >>> > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > PR here: >>> https://github.com/apache/airflow/pull/7832 >>> > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > J. >>> > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > On Tue, Mar 24, 2020 at 3:15 PM Kaxil Naik >>> < >>> > > > > kaxiln...@gmail.com >>> > > > > > > > > > > >>> > > > > > > > > > > > > > wrote: >>> > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > > @Tomasz Urbaszek < >>> tomasz.urbas...@polidea.com> : >>> > > > > > > > > > > > > > > > > Helm Chart Link: >>> > > > > https://github.com/astronomer/airflow-chart >>> > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > > On Tue, Mar 24, 2020 at 2:13 PM Tomasz >>> Urbaszek < >>> > > > > > > > > > > > > > turbas...@apache.org> >>> > > > > > > > > > > > > > > > > wrote: >>> > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > > > An official helm chart is something our >>> > > > > community needs! >>> > > > > > > > > > Using >>> > > > > > > > > > > > > your >>> > > > > > > > > > > > > > > > > > chart as the official makes a lot of >>> sens to me >>> > > > > because as >>> > > > > > > > > > you >>> > > > > > > > > > > > > > > > > > mentioned - it's battle tested. >>> > > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > > > One question: what Airflow image do >>> you use? >>> > > > > Also, would you >>> > > > > > > > > > > > > mind >>> > > > > > > > > > > > > > > > > > sharing a link to the chart? >>> > > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > > > Tomek >>> > > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > > > On Tue, Mar 24, 2020 at 2:07 PM Greg >>> Neiheisel >>> > > > > > > > > > > > > > > > > > <g...@astronomer.io.invalid> wrote: >>> > > > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > > > > Hey everyone, >>> > > > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > > > > Over the past few years at >>> Astronomer, we’ve >>> > > > > created, >>> > > > > > > > > > managed, >>> > > > > > > > > > > > > > and >>> > > > > > > > > > > > > > > > > > hardened >>> > > > > > > > > > > > > > > > > > > a production-ready Helm Chart for >>> Airflow ( >>> > > > > > > > > > > > > > > > > > > >>> https://github.com/astronomer/airflow-chart) >>> > > > > that is >>> > > > > > > > > > being >>> > > > > > > > > > > > > used >>> > > > > > > > > > > > > > by >>> > > > > > > > > > > > > > > > > both >>> > > > > > > > > > > > > > > > > > our >>> > > > > > > > > > > > > > > > > > > SaaS and Enterprise customers. This >>> chart is >>> > > > > > > > > > battle-tested and >>> > > > > > > > > > > > > > > running >>> > > > > > > > > > > > > > > > > > > hundreds of Airflow deployments of >>> varying >>> > > > > sizes and >>> > > > > > > > > > runtime >>> > > > > > > > > > > > > > > > > > environments. >>> > > > > > > > > > > > > > > > > > > It’s been built up to encapsulate >>> the issues >>> > > > > that Airflow >>> > > > > > > > > > > > > users >>> > > > > > > > > > > > > > run >>> > > > > > > > > > > > > > > > > into >>> > > > > > > > > > > > > > > > > > in >>> > > > > > > > > > > > > > > > > > > the real world. >>> > > > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > > > > While this chart was originally >>> developed >>> > > > > internally for >>> > > > > > > > > > our >>> > > > > > > > > > > > > > > Astronomer >>> > > > > > > > > > > > > > > > > > > Platform, we’ve recently decoupled >>> the chart >>> > > > > from the >>> > > > > > > > > > rest of >>> > > > > > > > > > > > > our >>> > > > > > > > > > > > > > > > > > platform >>> > > > > > > > > > > > > > > > > > > to make it usable by the greater >>> Airflow >>> > > > > community. With >>> > > > > > > > > > these >>> > > > > > > > > > > > > > > changes >>> > > > > > > > > > > > > > > > > in >>> > > > > > > > > > > > > > > > > > > mind, we want to start a >>> conversation about >>> > > > > donating this >>> > > > > > > > > > > > > chart >>> > > > > > > > > > > > > > to >>> > > > > > > > > > > > > > > the >>> > > > > > > > > > > > > > > > > > > Airflow community. >>> > > > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > > > > Some of the main features of the >>> chart are: >>> > > > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > > > > - It works out of the box. With zero >>> > > > > configuration, a user >>> > > > > > > > > > > > > will >>> > > > > > > > > > > > > > get >>> > > > > > > > > > > > > > > > > a >>> > > > > > > > > > > > > > > > > > > postgres database, a default user >>> and the >>> > > > > > > > > > KubernetesExecutor >>> > > > > > > > > > > > > > ready >>> > > > > > > > > > > > > > > > > to >>> > > > > > > > > > > > > > > > > > run >>> > > > > > > > > > > > > > > > > > > DAGs. >>> > > > > > > > > > > > > > > > > > > - Support for Local, Celery (w/ >>> optional KEDA >>> > > > > > > > > > autoscaling) and >>> > > > > > > > > > > > > > > > > > > Kubernetes executors. >>> > > > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > > > > Support for optional pgbouncer. We >>> use this to >>> > > > > share a >>> > > > > > > > > > > > > > configurable >>> > > > > > > > > > > > > > > > > > > connection pool size per deployment. >>> Useful >>> > > > > for limiting >>> > > > > > > > > > > > > > > connections to >>> > > > > > > > > > > > > > > > > > the >>> > > > > > > > > > > > > > > > > > > metadata database. >>> > > > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > > > > - Airflow migration support. A user >>> can push a >>> > > > > newer >>> > > > > > > > > > version >>> > > > > > > > > > > > > of >>> > > > > > > > > > > > > > > > > > Airflow >>> > > > > > > > > > > > > > > > > > > into an existing release and >>> migrations will >>> > > > > > > > > > automatically run >>> > > > > > > > > > > > > > > > > > cleanly. >>> > > > > > > > > > > > > > > > > > > - Prometheus support. Optionally >>> install and >>> > > > > configure a >>> > > > > > > > > > > > > > > > > > statsd-exporter >>> > > > > > > > > > > > > > > > > > > to ingest Airflow metrics and expose >>> them to >>> > > > > Prometheus >>> > > > > > > > > > > > > > > > > automatically. >>> > > > > > > > > > > > > > > > > > > - Resource control. Optionally >>> control the >>> > > > > ResourceQuotas >>> > > > > > > > > > and >>> > > > > > > > > > > > > > > > > > > LimitRanges for each deployment so >>> that no >>> > > > > deployment can >>> > > > > > > > > > > > > > overload >>> > > > > > > > > > > > > > > a >>> > > > > > > > > > > > > > > > > > > cluster. >>> > > > > > > > > > > > > > > > > > > - Simple optional Elasticsearch >>> support. >>> > > > > > > > > > > > > > > > > > > - Optional namespace cleanup. >>> Sometimes >>> > > > > > > > > > KubernetesExecutor and >>> > > > > > > > > > > > > > > > > > > KubernetesPodOperator pods fail for >>> reasons >>> > > > > other than the >>> > > > > > > > > > > > > actual >>> > > > > > > > > > > > > > > > > > task. >>> > > > > > > > > > > > > > > > > > > This feature helps keep things clean >>> in >>> > > > > Kubernetes. >>> > > > > > > > > > > > > > > > > > > - Support for running locally in KIND >>> > > > > (Kubernetes in >>> > > > > > > > > > Docker). >>> > > > > > > > > > > > > > > > > > > - Automatically tested across many >>> Kubernetes >>> > > > > versions >>> > > > > > > > > > with >>> > > > > > > > > > > > > Helm >>> > > > > > > > > > > > > > 2 >>> > > > > > > > > > > > > > > > > > and 3 >>> > > > > > > > > > > > > > > > > > > support. >>> > > > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > > > > We’ve found that the cleanest and >>> most >>> > > > > reliable way to >>> > > > > > > > > > deploy >>> > > > > > > > > > > > > > DAGs >>> > > > > > > > > > > > > > > to >>> > > > > > > > > > > > > > > > > > > Kubernetes and manage them at scale >>> is to >>> > > > > package them >>> > > > > > > > > > into >>> > > > > > > > > > > > > the >>> > > > > > > > > > > > > > > actual >>> > > > > > > > > > > > > > > > > > > docker image, so we have geared this >>> chart >>> > > > > towards that >>> > > > > > > > > > > > > method of >>> > > > > > > > > > > > > > > > > > > operation, though adding other >>> methods should >>> > > > > be >>> > > > > > > > > > > > > straightforward. >>> > > > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > > > > We would love thoughts from the >>> community and >>> > > > > would love >>> > > > > > > > > > to >>> > > > > > > > > > > > > see >>> > > > > > > > > > > > > > > this >>> > > > > > > > > > > > > > > > > > chart >>> > > > > > > > > > > > > > > > > > > help others to get up and running on >>> > > > > Kubernetes! >>> > > > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > > > > -- >>> > > > > > > > > > > > > > > > > > > *Greg Neiheisel* / Chief Architect >>> > > > > Astronomer.io >>> > > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > -- >>> > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > Jarek Potiuk >>> > > > > > > > > > > > > > > > Polidea <https://www.polidea.com/> | >>> Principal >>> > > > > Software >>> > > > > > > > > > Engineer >>> > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > M: +48 660 796 129 <+48%20660%20796%20129> >>> > > > > <+48660796129 >>> > > > > > > > > > > > > <+48%20660%20796%20129>> >>> > > > > > > > > > > > > > > > [image: Polidea] <https://www.polidea.com/ >>> > >>> > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > >>> > > > > > > > > > > > > > >>> > > > > > > > > > > > > > -- >>> > > > > > > > > > > > > > *Greg Neiheisel* / Chief Architect >>> Astronomer.io >>> > > > > > > > > > > > > > >>> > > > > > > > > > > > > >>> > > > > > > > > > > > > >>> > > > > > > > > > > > > -- >>> > > > > > > > > > > > > >>> > > > > > > > > > > > > Jarek Potiuk >>> > > > > > > > > > > > > Polidea <https://www.polidea.com/> | Principal >>> Software >>> > > > > Engineer >>> > > > > > > > > > > > > >>> > > > > > > > > > > > > M: +48 660 796 129 <+48%20660%20796%20129> >>> <+48660796129 >>> > > > > > > > > > > > > <+48%20660%20796%20129>> >>> > > > > > > > > > > > > [image: Polidea] <https://www.polidea.com/> >>> > > > > > > > > > > > > >>> > > > > > > > > > > > >>> > > > > > > > > > > > >>> > > > > > > > > > > > -- >>> > > > > > > > > > > > *Greg Neiheisel* / Chief Architect Astronomer.io >>> > > > > > > > > > > > >>> > > > > > > > > > >>> > > > > > > > > > >>> > > > > > > > > > >>> > > > > > > > > > -- >>> > > > > > > > > > >>> > > > > > > > > > Jarek Potiuk >>> > > > > > > > > > Polidea | Principal Software Engineer >>> > > > > > > > > > >>> > > > > > > > > > M: +48 660 796 129 >>> > > > > > > > > > >>> > > > > > > > > >>> > > > > > > > > >>> > > > > > > > > -- >>> > > > > > > > > *Greg Neiheisel* / Chief Architect Astronomer.io >>> > > > > > >>> > > > > > >>> > > > > > -- >>> > > > > > Jarek Potiuk >>> > > > > > Polidea | Principal Software Engineer >>> > > > > > M: +48 660 796 129 >>> > > > > > >>> > > > > >>> > >>> > >>> > >>> > -- >>> > >>> > Jarek Potiuk >>> > Polidea | Principal Software Engineer >>> > >>> > M: +48 660 796 129 >>> >> >> >> -- >> >> Jarek Potiuk >> Polidea <https://www.polidea.com/> | Principal Software Engineer >> >> M: +48 660 796 129 <+48660796129> >> [image: Polidea] <https://www.polidea.com/> >> >> > > -- > > Jarek Potiuk > Polidea <https://www.polidea.com/> | Principal Software Engineer > > M: +48 660 796 129 <+48660796129> > [image: Polidea] <https://www.polidea.com/> > > -- Jarek Potiuk Polidea <https://www.polidea.com/> | Principal Software Engineer M: +48 660 796 129 <+48660796129> [image: Polidea] <https://www.polidea.com/>
