Miretpl commented on code in PR #65897: URL: https://github.com/apache/airflow/pull/65897#discussion_r3164367051
########## chart/kustomize-overlays/keda/README.rst: ########## @@ -0,0 +1,183 @@ +.. Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + +.. http://www.apache.org/licenses/LICENSE-2.0 + +.. Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. + +KEDA Autoscaler Overlay +======================= + +This overlay produces a `KEDA <https://keda.sh/>`__ ``ScaledObject`` plus a +``TriggerAuthentication`` for the chart-rendered Celery workers. It is a +standalone addition: no resource produced by the Helm chart is modified. + +It is the Kustomize equivalent of enabling ``workers.celery.keda.enabled`` +in ``values.yaml``, and is the recommended migration path for users who +want to keep Celery autoscaling without relying on chart-side templating. + +Prerequisites +------------- + +* `KEDA <https://keda.sh/docs/latest/deploy/>`__ installed in the cluster. +* The Airflow chart installed with the **CeleryExecutor** (or + CeleryKubernetesExecutor). +* The chart's metadata Secret (``<release>-airflow-metadata``) reachable + from KEDA's namespace - usually the same namespace as the chart release. + +Resources produced +------------------ + +* ``TriggerAuthentication/airflow-keda-postgres-auth`` - reads the metadata + DB connection string directly from the chart's metadata Secret. +* ``ScaledObject/airflow-worker`` - targets the chart-rendered worker + Deployment and scales it based on the count of running and queued task + instances. + +Usage +----- + +Reference this overlay from your own kustomization and substitute the +release name. A minimal example: + +.. code-block:: yaml + + # my-overlay/kustomization.yaml + apiVersion: kustomize.config.k8s.io/v1beta1 + kind: Kustomization + namespace: airflow + + resources: + - github.com/apache/airflow/chart/kustomize-overlays/keda?ref=helm-chart/1.21.0 + + replacements: + - source: + kind: ConfigMap + name: airflow-overlay-config + fieldPath: data.releaseName + targets: + - select: + kind: ScaledObject + name: airflow-worker + fieldPaths: + - spec.scaleTargetRef.name + options: + delimiter: "-" + index: 0 + - select: + kind: TriggerAuthentication + name: airflow-keda-postgres-auth + fieldPaths: + - spec.secretTargetRef.0.name + options: + delimiter: "-" + index: 0 + + configMapGenerator: + - name: airflow-overlay-config + literals: + - releaseName=airflow + +Apply with: + +.. code-block:: bash + + kubectl apply -k my-overlay/ + +For a quick test, you can also just ``sed`` the placeholder: + +.. code-block:: bash + + kustomize build chart/kustomize-overlays/keda | \ + sed 's/RELEASE-NAME/airflow/g' | \ + kubectl apply -f - + +Tuning the trigger query +------------------------ + +The default query mirrors the chart for a single Celery queue named +``default`` with ``worker_concurrency=16``. If you set different values in +your chart install, edit ``scaledobject.yaml`` accordingly: + +* Replace ``16`` with the value of ``config.celery.worker_concurrency``. +* Extend ``queue IN ('default')`` to list every entry from + ``workers.queue`` (comma-separated in ``values.yaml``, single-quoted Review Comment: ```suggestion ``workers.celery.queue`` (comma-separated in ``values.yaml``, single-quoted ``` ########## chart/kustomize-overlays/keda/README.rst: ########## @@ -0,0 +1,183 @@ +.. Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + +.. http://www.apache.org/licenses/LICENSE-2.0 + +.. Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. + +KEDA Autoscaler Overlay +======================= + +This overlay produces a `KEDA <https://keda.sh/>`__ ``ScaledObject`` plus a +``TriggerAuthentication`` for the chart-rendered Celery workers. It is a +standalone addition: no resource produced by the Helm chart is modified. + +It is the Kustomize equivalent of enabling ``workers.celery.keda.enabled`` +in ``values.yaml``, and is the recommended migration path for users who +want to keep Celery autoscaling without relying on chart-side templating. + +Prerequisites +------------- + +* `KEDA <https://keda.sh/docs/latest/deploy/>`__ installed in the cluster. +* The Airflow chart installed with the **CeleryExecutor** (or + CeleryKubernetesExecutor). +* The chart's metadata Secret (``<release>-airflow-metadata``) reachable + from KEDA's namespace - usually the same namespace as the chart release. + +Resources produced +------------------ + +* ``TriggerAuthentication/airflow-keda-postgres-auth`` - reads the metadata + DB connection string directly from the chart's metadata Secret. +* ``ScaledObject/airflow-worker`` - targets the chart-rendered worker + Deployment and scales it based on the count of running and queued task + instances. + +Usage +----- + +Reference this overlay from your own kustomization and substitute the +release name. A minimal example: + +.. code-block:: yaml + + # my-overlay/kustomization.yaml + apiVersion: kustomize.config.k8s.io/v1beta1 + kind: Kustomization + namespace: airflow + + resources: + - github.com/apache/airflow/chart/kustomize-overlays/keda?ref=helm-chart/1.21.0 + + replacements: + - source: + kind: ConfigMap + name: airflow-overlay-config + fieldPath: data.releaseName + targets: + - select: + kind: ScaledObject + name: airflow-worker + fieldPaths: + - spec.scaleTargetRef.name + options: + delimiter: "-" + index: 0 + - select: + kind: TriggerAuthentication + name: airflow-keda-postgres-auth + fieldPaths: + - spec.secretTargetRef.0.name + options: + delimiter: "-" + index: 0 + + configMapGenerator: + - name: airflow-overlay-config + literals: + - releaseName=airflow + +Apply with: + +.. code-block:: bash + + kubectl apply -k my-overlay/ + +For a quick test, you can also just ``sed`` the placeholder: + +.. code-block:: bash + + kustomize build chart/kustomize-overlays/keda | \ + sed 's/RELEASE-NAME/airflow/g' | \ + kubectl apply -f - + +Tuning the trigger query +------------------------ + +The default query mirrors the chart for a single Celery queue named +``default`` with ``worker_concurrency=16``. If you set different values in +your chart install, edit ``scaledobject.yaml`` accordingly: + +* Replace ``16`` with the value of ``config.celery.worker_concurrency``. +* Extend ``queue IN ('default')`` to list every entry from + ``workers.queue`` (comma-separated in ``values.yaml``, single-quoted + here). +* For ``CeleryKubernetesExecutor`` add + ``AND queue != '<config.celery_kubernetes_executor.kubernetes_queue>'``. Review Comment: ```suggestion ``` Removed with support for Airflow 2. ########## chart/kustomize-overlays/README.rst: ########## @@ -0,0 +1,71 @@ +.. Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + +.. http://www.apache.org/licenses/LICENSE-2.0 + +.. Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. + +Airflow Helm Chart - Kustomize Overlays +======================================= + +.. note:: + + **Not distributed with chart releases.** + This directory lives in the source repository as a reference for users but + is **not** packaged or published as part of the official Airflow Helm chart + release artifacts. Consume it directly from the repository at the tag that + matches your chart version. + +This directory contains Kustomize overlays that complement the Airflow Helm +chart for components that are not Airflow-native. + +The motivation, criteria, and lifecycle for these overlays are defined in +``CONTRIBUTING.rst`` in this directory. + +Available overlays +------------------ + ++----------+----------------------+----------------------------------------------+ +| Overlay | STATUS | Purpose | ++==========+======================+==============================================+ +| ``keda`` | not-tested (PoC) | Autoscaling for Celery workers via KEDA. | ++----------+----------------------+----------------------------------------------+ + +Each overlay directory contains its own ``README.rst`` with usage details and +a migration guide from the equivalent chart-side configuration. + +Using an overlay +---------------- + +The overlays are designed for the "standalone addition" pattern. They do not +modify resources rendered by the chart. A typical workflow is: + +1. Install the Airflow chart as usual. +2. Reference the overlay from your own ``kustomization.yaml`` and apply the + substitutions described in the overlay's ``README.rst`` (release name, + namespace, secret references). +3. Apply the rendered manifests with ``kubectl apply -k`` against the same + namespace as the chart release. + +Status conventions +------------------ + +Each overlay carries a ``STATUS.yaml`` file that declares its verification level: + +* ``tested`` - the overlay is verified in CI against the current chart version. +* ``not-tested`` - the overlay builds successfully but has no functional CI + coverage. Treat it as a starting point that you adapt to your environment. +* ``deprecated`` - the overlay is scheduled for removal. The ``STATUS.yaml`` file + carries a ``message`` field pointing to the replacement. + +See ``CONTRIBUTING.rst`` for the full status grammar and lifecycle. Review Comment: Could we make a `CONTRIBUTING.rst` a link? ########## chart/kustomize-overlays/README.rst: ########## @@ -0,0 +1,71 @@ +.. Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + +.. http://www.apache.org/licenses/LICENSE-2.0 + +.. Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. + +Airflow Helm Chart - Kustomize Overlays +======================================= + +.. note:: + + **Not distributed with chart releases.** + This directory lives in the source repository as a reference for users but + is **not** packaged or published as part of the official Airflow Helm chart + release artifacts. Consume it directly from the repository at the tag that + matches your chart version. + +This directory contains Kustomize overlays that complement the Airflow Helm +chart for components that are not Airflow-native. + +The motivation, criteria, and lifecycle for these overlays are defined in +``CONTRIBUTING.rst`` in this directory. + +Available overlays +------------------ + ++----------+----------------------+----------------------------------------------+ +| Overlay | STATUS | Purpose | ++==========+======================+==============================================+ +| ``keda`` | not-tested (PoC) | Autoscaling for Celery workers via KEDA. | ++----------+----------------------+----------------------------------------------+ + +Each overlay directory contains its own ``README.rst`` with usage details and +a migration guide from the equivalent chart-side configuration. + +Using an overlay +---------------- + +The overlays are designed for the "standalone addition" pattern. They do not +modify resources rendered by the chart. A typical workflow is: + +1. Install the Airflow chart as usual. +2. Reference the overlay from your own ``kustomization.yaml`` and apply the + substitutions described in the overlay's ``README.rst`` (release name, + namespace, secret references). +3. Apply the rendered manifests with ``kubectl apply -k`` against the same + namespace as the chart release. + +Status conventions +------------------ + +Each overlay carries a ``STATUS.yaml`` file that declares its verification level: + +* ``tested`` - the overlay is verified in CI against the current chart version. Review Comment: ```suggestion * ``tested`` - the overlay is verified in Apache Airflow CI against the current chart version. ``` ########## chart/kustomize-overlays/keda/README.rst: ########## @@ -0,0 +1,183 @@ +.. Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + +.. http://www.apache.org/licenses/LICENSE-2.0 + +.. Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. + +KEDA Autoscaler Overlay +======================= + +This overlay produces a `KEDA <https://keda.sh/>`__ ``ScaledObject`` plus a +``TriggerAuthentication`` for the chart-rendered Celery workers. It is a +standalone addition: no resource produced by the Helm chart is modified. + +It is the Kustomize equivalent of enabling ``workers.celery.keda.enabled`` +in ``values.yaml``, and is the recommended migration path for users who +want to keep Celery autoscaling without relying on chart-side templating. + +Prerequisites +------------- + +* `KEDA <https://keda.sh/docs/latest/deploy/>`__ installed in the cluster. +* The Airflow chart installed with the **CeleryExecutor** (or + CeleryKubernetesExecutor). Review Comment: ```suggestion * The Airflow chart installed with the **CeleryExecutor**. ``` We removed support for Airflow 2, where this executor was supported. ########## chart/kustomize-overlays/keda/README.rst: ########## @@ -0,0 +1,183 @@ +.. Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + +.. http://www.apache.org/licenses/LICENSE-2.0 + +.. Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. + +KEDA Autoscaler Overlay +======================= + +This overlay produces a `KEDA <https://keda.sh/>`__ ``ScaledObject`` plus a +``TriggerAuthentication`` for the chart-rendered Celery workers. It is a +standalone addition: no resource produced by the Helm chart is modified. + +It is the Kustomize equivalent of enabling ``workers.celery.keda.enabled`` +in ``values.yaml``, and is the recommended migration path for users who +want to keep Celery autoscaling without relying on chart-side templating. + +Prerequisites +------------- + +* `KEDA <https://keda.sh/docs/latest/deploy/>`__ installed in the cluster. +* The Airflow chart installed with the **CeleryExecutor** (or + CeleryKubernetesExecutor). +* The chart's metadata Secret (``<release>-airflow-metadata``) reachable + from KEDA's namespace - usually the same namespace as the chart release. + +Resources produced +------------------ + +* ``TriggerAuthentication/airflow-keda-postgres-auth`` - reads the metadata + DB connection string directly from the chart's metadata Secret. +* ``ScaledObject/airflow-worker`` - targets the chart-rendered worker + Deployment and scales it based on the count of running and queued task + instances. + +Usage +----- + +Reference this overlay from your own kustomization and substitute the +release name. A minimal example: + +.. code-block:: yaml + + # my-overlay/kustomization.yaml + apiVersion: kustomize.config.k8s.io/v1beta1 + kind: Kustomization + namespace: airflow + + resources: + - github.com/apache/airflow/chart/kustomize-overlays/keda?ref=helm-chart/1.21.0 + + replacements: + - source: + kind: ConfigMap + name: airflow-overlay-config + fieldPath: data.releaseName + targets: + - select: + kind: ScaledObject + name: airflow-worker + fieldPaths: + - spec.scaleTargetRef.name + options: + delimiter: "-" + index: 0 + - select: + kind: TriggerAuthentication + name: airflow-keda-postgres-auth + fieldPaths: + - spec.secretTargetRef.0.name + options: + delimiter: "-" + index: 0 + + configMapGenerator: + - name: airflow-overlay-config + literals: + - releaseName=airflow + +Apply with: + +.. code-block:: bash + + kubectl apply -k my-overlay/ + +For a quick test, you can also just ``sed`` the placeholder: + +.. code-block:: bash + + kustomize build chart/kustomize-overlays/keda | \ + sed 's/RELEASE-NAME/airflow/g' | \ + kubectl apply -f - + +Tuning the trigger query +------------------------ + +The default query mirrors the chart for a single Celery queue named +``default`` with ``worker_concurrency=16``. If you set different values in +your chart install, edit ``scaledobject.yaml`` accordingly: + +* Replace ``16`` with the value of ``config.celery.worker_concurrency``. +* Extend ``queue IN ('default')`` to list every entry from + ``workers.queue`` (comma-separated in ``values.yaml``, single-quoted + here). +* For ``CeleryKubernetesExecutor`` add + ``AND queue != '<config.celery_kubernetes_executor.kubernetes_queue>'``. + +Pgbouncer +--------- + +If pgbouncer is enabled and you do not want KEDA polling through it, change +the ``key`` field in ``triggerauthentication.yaml`` from ``connection`` to +``kedaConnection``. The chart writes a direct-to-Postgres connection string +under that key for exactly this purpose. + +Persistence +----------- + +If your worker is deployed as a ``StatefulSet`` (i.e. you set +``workers.persistence.enabled=true``), change ``kind: Deployment`` to Review Comment: ```suggestion ``workers.celery.persistence.enabled=true``), change ``kind: Deployment`` to ``` ########## chart/kustomize-overlays/CONTRIBUTING.rst: ########## @@ -0,0 +1,158 @@ +.. Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + +.. http://www.apache.org/licenses/LICENSE-2.0 + +.. Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. + +Contributing Kustomize Overlays +=============================== + +This document is the authoritative reference for adding, evolving, and +retiring overlays under ``chart/kustomize-overlays/``. + +Why this directory exists +------------------------- + +The Airflow Helm chart has historically carried components that are not +Airflow-native. They make the chart heavier than it needs to be and pull +maintenance toward things that already have external owners. Expressing +these components as Kustomize overlays keeps the chart focused on Airflow +itself while still giving users a working starting point for the rest. + +The chart never removes a component without a working overlay already in +place. Users always have a migration path before anything disappears. + +Criteria for chart vs Kustomize +------------------------------- + +A component **belongs in the chart** when all of the following are true: + +* It is required to run Airflow (scheduler, API server, dag-processor, + triggerer, workers). +* Removing it requires changes to Airflow's own configuration. +* It has no external owner. +* It is assumed that the larger majority (>80%) will need and use this function for productive use + +A component **belongs in Kustomize** when any of the following are true: + +* It can be expressed as a standalone Kubernetes resource without modifying + chart-rendered resources. +* It is environment-specific (authentication schemes, logging backends, + autoscaling controllers). +* It has an external owner (KEDA, Elasticsearch, any PostgreSQL distribution). Review Comment: ```suggestion autoscaling controllers, etc.). * It has an external owner (KEDA, Elasticsearch, any PostgreSQL distribution, etc.). ``` ########## chart/kustomize-overlays/CONTRIBUTING.rst: ########## @@ -0,0 +1,158 @@ +.. Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + +.. http://www.apache.org/licenses/LICENSE-2.0 + +.. Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. + +Contributing Kustomize Overlays +=============================== + +This document is the authoritative reference for adding, evolving, and +retiring overlays under ``chart/kustomize-overlays/``. + +Why this directory exists +------------------------- + +The Airflow Helm chart has historically carried components that are not +Airflow-native. They make the chart heavier than it needs to be and pull +maintenance toward things that already have external owners. Expressing +these components as Kustomize overlays keeps the chart focused on Airflow +itself while still giving users a working starting point for the rest. + +The chart never removes a component without a working overlay already in +place. Users always have a migration path before anything disappears. + +Criteria for chart vs Kustomize +------------------------------- + +A component **belongs in the chart** when all of the following are true: + +* It is required to run Airflow (scheduler, API server, dag-processor, + triggerer, workers). +* Removing it requires changes to Airflow's own configuration. Review Comment: ```suggestion * Removing/adding it requires changes to Airflow's own configuration. ``` to include new features too ########## chart/kustomize-overlays/README.rst: ########## @@ -0,0 +1,71 @@ +.. Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + +.. http://www.apache.org/licenses/LICENSE-2.0 + +.. Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. + +Airflow Helm Chart - Kustomize Overlays +======================================= + +.. note:: + + **Not distributed with chart releases.** + This directory lives in the source repository as a reference for users but + is **not** packaged or published as part of the official Airflow Helm chart + release artifacts. Consume it directly from the repository at the tag that + matches your chart version. + +This directory contains Kustomize overlays that complement the Airflow Helm +chart for components that are not Airflow-native. + +The motivation, criteria, and lifecycle for these overlays are defined in +``CONTRIBUTING.rst`` in this directory. + +Available overlays +------------------ + ++----------+----------------------+----------------------------------------------+ +| Overlay | STATUS | Purpose | ++==========+======================+==============================================+ +| ``keda`` | not-tested (PoC) | Autoscaling for Celery workers via KEDA. | ++----------+----------------------+----------------------------------------------+ Review Comment: Just for the future: it would be nice to have some automation for generating that table ########## chart/kustomize-overlays/CONTRIBUTING.rst: ########## @@ -0,0 +1,158 @@ +.. Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + +.. http://www.apache.org/licenses/LICENSE-2.0 + +.. Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. + +Contributing Kustomize Overlays +=============================== + +This document is the authoritative reference for adding, evolving, and +retiring overlays under ``chart/kustomize-overlays/``. + +Why this directory exists +------------------------- + +The Airflow Helm chart has historically carried components that are not +Airflow-native. They make the chart heavier than it needs to be and pull +maintenance toward things that already have external owners. Expressing +these components as Kustomize overlays keeps the chart focused on Airflow +itself while still giving users a working starting point for the rest. + +The chart never removes a component without a working overlay already in +place. Users always have a migration path before anything disappears. + +Criteria for chart vs Kustomize +------------------------------- + +A component **belongs in the chart** when all of the following are true: + +* It is required to run Airflow (scheduler, API server, dag-processor, + triggerer, workers). +* Removing it requires changes to Airflow's own configuration. +* It has no external owner. +* It is assumed that the larger majority (>80%) will need and use this function for productive use + +A component **belongs in Kustomize** when any of the following are true: + +* It can be expressed as a standalone Kubernetes resource without modifying + chart-rendered resources. +* It is environment-specific (authentication schemes, logging backends, + autoscaling controllers). +* It has an external owner (KEDA, Elasticsearch, any PostgreSQL distribution). +* It requires CRDs that the chart does not install. +* It is used by only a few users such that additional complexity and maintenance burden does not pay out Review Comment: ```suggestion * It is used by a minority of users, such that the additional complexity and maintenance burden do not pay off ``` I think that sounds a little better ########## chart/kustomize-overlays/CONTRIBUTING.rst: ########## @@ -0,0 +1,158 @@ +.. Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + +.. http://www.apache.org/licenses/LICENSE-2.0 + +.. Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. + +Contributing Kustomize Overlays +=============================== + +This document is the authoritative reference for adding, evolving, and +retiring overlays under ``chart/kustomize-overlays/``. + +Why this directory exists +------------------------- + +The Airflow Helm chart has historically carried components that are not +Airflow-native. They make the chart heavier than it needs to be and pull +maintenance toward things that already have external owners. Expressing +these components as Kustomize overlays keeps the chart focused on Airflow +itself while still giving users a working starting point for the rest. + +The chart never removes a component without a working overlay already in +place. Users always have a migration path before anything disappears. + +Criteria for chart vs Kustomize +------------------------------- + +A component **belongs in the chart** when all of the following are true: + +* It is required to run Airflow (scheduler, API server, dag-processor, + triggerer, workers). +* Removing it requires changes to Airflow's own configuration. +* It has no external owner. +* It is assumed that the larger majority (>80%) will need and use this function for productive use Review Comment: ```suggestion * It is used by the larger majority of users (>80%) for production-ready environments ``` Maybe we should also determine it by the possible usage of the feature in terms of where it will be used (prod or non-prod env)? WDYT? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
