jedcunningham commented on code in PR #62544: URL: https://github.com/apache/airflow/pull/62544#discussion_r2875113789
########## chart/docs/keda.rst: ########## @@ -18,63 +18,65 @@ Autoscaling with KEDA ===================== -*This feature is still experimental.* - KEDA stands for Kubernetes Event Driven Autoscaling. `KEDA <https://github.com/kedacore/keda>`__ is a custom controller that allows users to create custom bindings to the Kubernetes `Horizontal Pod Autoscaler <https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/>`__. The autoscaler will adjust the number of active Celery workers based on the number of tasks in ``queued`` or ``running`` state. -One advantage of KEDA is it allows you to scale your application to/from 0 workers, meaning no workers are idle when there are no tasks. +One advantage of KEDA is that it allows you to scale your application to/from 0 workers, meaning no workers are idle when there are no tasks. + +KEDA Installation and usage +--------------------------- + +To install KEDA in your Kubernetes cluster, run the following commands: .. code-block:: bash helm repo add kedacore https://kedacore.github.io/charts - helm repo update - kubectl create namespace keda - helm install keda kedacore/keda \ - --namespace keda \ - --version "v2.0.0" - -Enable for the Airflow instance by setting ``workers.celery.keda.enabled=true`` in your -helm command or in the ``values.yaml``. + --namespace keda \ + --version "v2.0.0" -Make sure ``values.yaml`` shows that either KEDA or HPA is enabled, but not both. - -It is recommended not to use both a KEDA and an HPA to scale the same workload. -They will compete with each other resulting in odd scaling behavior. +To enable KEDA for the Airflow instance, it has to be enabled by setting ``workers.celery.keda.enabled=true`` +in your Helm command or in the ``values.yaml`` like: .. code-block:: bash kubectl create namespace airflow helm repo add apache-airflow https://airflow.apache.org helm install airflow apache-airflow/airflow \ - --namespace airflow \ - --set executor=CeleryExecutor \ - --set workers.celery.keda.enabled=true + --namespace airflow \ + --set executor=CeleryExecutor \ + --set workers.celery.keda.enabled=true + +.. note:: + + Make sure ``values.yaml`` shows that either KEDA or HPA is enabled, but not both. It is recommended not + to use both KEDA and HPA to scale the same workload. They will compete with each other resulting in odd scaling behavior. -A ``ScaledObject`` and an ``hpa`` will be created in the Airflow namespace. +After installation, the KEDA ``ScaledObject`` and an ``HPA`` will be created in the Airflow namespace. -KEDA will derive the desired number of Celery workers by querying -Airflow metadata database: +In the default configuration, KEDA will derive the desired number of Celery workers by querying Airflow metadata database with following SQL statement: .. code-block:: none SELECT - ceil(COUNT(*)::decimal / {{ .Values.config.celery.worker_concurrency }}) - FROM task_instance - WHERE state='running' OR state='queued' + ceil(COUNT(*)::decimal / {{ .Values.config.celery.worker_concurrency }}) + FROM + task_instance + WHERE + state='running' OR state='queued' + AND queue IN ('default') Review Comment: Looks good, thanks! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
