Hey everyone, Over the past few years at Astronomer, we’ve created, managed, and hardened a production-ready Helm Chart for Airflow ( https://github.com/astronomer/airflow-chart) that is being used by both our SaaS and Enterprise customers. This chart is battle-tested and running hundreds of Airflow deployments of varying sizes and runtime environments. It’s been built up to encapsulate the issues that Airflow users run into in the real world.
While this chart was originally developed internally for our Astronomer Platform, we’ve recently decoupled the chart from the rest of our platform to make it usable by the greater Airflow community. With these changes in mind, we want to start a conversation about donating this chart to the Airflow community. Some of the main features of the chart are: - It works out of the box. With zero configuration, a user will get a postgres database, a default user and the KubernetesExecutor ready to run DAGs. - Support for Local, Celery (w/ optional KEDA autoscaling) and Kubernetes executors. Support for optional pgbouncer. We use this to share a configurable connection pool size per deployment. Useful for limiting connections to the metadata database. - Airflow migration support. A user can push a newer version of Airflow into an existing release and migrations will automatically run cleanly. - Prometheus support. Optionally install and configure a statsd-exporter to ingest Airflow metrics and expose them to Prometheus automatically. - Resource control. Optionally control the ResourceQuotas and LimitRanges for each deployment so that no deployment can overload a cluster. - Simple optional Elasticsearch support. - Optional namespace cleanup. Sometimes KubernetesExecutor and KubernetesPodOperator pods fail for reasons other than the actual task. This feature helps keep things clean in Kubernetes. - Support for running locally in KIND (Kubernetes in Docker). - Automatically tested across many Kubernetes versions with Helm 2 and 3 support. We’ve found that the cleanest and most reliable way to deploy DAGs to Kubernetes and manage them at scale is to package them into the actual docker image, so we have geared this chart towards that method of operation, though adding other methods should be straightforward. We would love thoughts from the community and would love to see this chart help others to get up and running on Kubernetes! -- *Greg Neiheisel* / Chief Architect Astronomer.io
