asosnovsky-sumologic commented on code in PR #30914:
URL: https://github.com/apache/airflow/pull/30914#discussion_r1181822209
##########
chart/values.yaml:
##########
@@ -704,6 +718,51 @@ scheduler:
# Annotations to add to scheduler kubernetes service account.
annotations: {}
+
+ # Allow KEDA autoscaling.
+ keda:
+ enabled: false
+ namespaceLabels: {}
+
+ # How often KEDA polls the airflow DB to report new scale requests to the
HPA
+ pollingInterval: 5
+
+ # How many seconds KEDA will wait before scaling to zero.
+ # Note that HPA has a separate cooldown period for scale-downs
+ cooldownPeriod: 30
+
+ # Minimum number of workers created by keda
+ minReplicaCount: 0
+
+ # Maximum number of workers created by keda
+ maxReplicaCount: 10
+
+ # Specify HPA related options
+ advanced: {}
+ # horizontalPodAutoscalerConfig:
+ # behavior:
+ # scaleDown:
+ # stabilizationWindowSeconds: 300
+ # policies:
+ # - type: Percent
+ # value: 100
+ # periodSeconds: 15
+
+ # Specify the KEDA Triggers for autoscaling
+ triggers: >-
+ - type: postgresql
+ metadata:
+ targetQueryValue: "1"
+ connectionFromEnv: AIRFLOW_CONN_AIRFLOW_DB
+ query: >-
+ SELECT ceil(COUNT(*)::decimal / {{ .Values.config.core.parallelism
}})
+ FROM task_instance
+ WHERE (state='running' OR state='queued')
+ {{- if eq .Values.executor "CeleryKubernetesExecutor" }}
+ AND queue != '{{
.Values.config.celery_kubernetes_executor.kubernetes_queue }}'
+ {{- end }}
Review Comment:
because `parallelism` determines how many tasks a single scheduler will
manage
https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#parallelism
the rest is more or less mimicing what the worker query does
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
