dstandish commented on code in PR #30914:
URL: https://github.com/apache/airflow/pull/30914#discussion_r1181824888
##########
chart/values.yaml:
##########
@@ -704,6 +718,51 @@ scheduler:
# Annotations to add to scheduler kubernetes service account.
annotations: {}
+
+ # Allow KEDA autoscaling.
+ keda:
+ enabled: false
+ namespaceLabels: {}
+
+ # How often KEDA polls the airflow DB to report new scale requests to the
HPA
+ pollingInterval: 5
+
+ # How many seconds KEDA will wait before scaling to zero.
+ # Note that HPA has a separate cooldown period for scale-downs
+ cooldownPeriod: 30
+
+ # Minimum number of workers created by keda
+ minReplicaCount: 0
+
+ # Maximum number of workers created by keda
+ maxReplicaCount: 10
+
+ # Specify HPA related options
+ advanced: {}
+ # horizontalPodAutoscalerConfig:
+ # behavior:
+ # scaleDown:
+ # stabilizationWindowSeconds: 300
+ # policies:
+ # - type: Percent
+ # value: 100
+ # periodSeconds: 15
+
+ # Specify the KEDA Triggers for autoscaling
+ triggers: >-
+ - type: postgresql
+ metadata:
+ targetQueryValue: "1"
+ connectionFromEnv: AIRFLOW_CONN_AIRFLOW_DB
+ query: >-
+ SELECT ceil(COUNT(*)::decimal / {{ .Values.config.core.parallelism
}})
+ FROM task_instance
+ WHERE (state='running' OR state='queued')
+ {{- if eq .Values.executor "CeleryKubernetesExecutor" }}
+ AND queue != '{{
.Values.config.celery_kubernetes_executor.kubernetes_queue }}'
+ {{- end }}
Review Comment:
The worker query makes sense because all tasks need workers (except those
running on kubernetes hence the case for that)
However, not all tasks are deferrable. So I think you would want to look
not at overall queued tasks, but tasks i think in a deferred state which have
not been picked up by a triggerer. Then you'd want to compare that with
triggerer concurrency not overall parallelism I believe.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
