aaron-y-chen commented on code in PR #75220:
URL: https://github.com/apache/airflow/pull/75220#discussion_r4367944235
##########
chart/templates/workers/worker-kedaautoscaler.yaml:
##########
@@ -60,19 +60,29 @@ spec:
triggers:
{{- if eq .Values.data.metadataConnection.protocol "mysql" }}
- type: "mysql"
+ {{- with .Values.workers.celery.keda.authenticationRef }}
+ authenticationRef: {{- toYaml . | nindent 8 }}
+ {{- end }}
metadata:
queryValue: "1"
+ {{- if not .Values.workers.celery.keda.authenticationRef }}
connectionStringFromEnv: KEDA_DB_CONN
+ {{- end }}
query: {{ tpl .Values.workers.celery.keda.query . | quote }}
{{- else }}
- type: "postgresql"
+ {{- with .Values.workers.celery.keda.authenticationRef }}
+ authenticationRef: {{- toYaml . | nindent 8 }}
+ {{- end }}
metadata:
targetQueryValue: "1"
+ {{- if not .Values.workers.celery.keda.authenticationRef }}
Review Comment:
With `authenticationRef` set, the `ScaledObject` does not read the
connection from the worker environment. But the worker still gets
`KEDA_DB_CONN`. This occurs for MySQL, and for PostgreSQL when PgBouncer is
enabled and `usePgbouncer` is false.
The cause is in `chart/templates/workers/worker-deployment.yaml`. This
template includes `keda_airflow_environment` when KEDA is enabled. It does not
examine `authenticationRef`. Thus, the worker pod still has a required
reference to the `kedaConnection` key in the metadata Secret.
This has two results:
- The worker pod has a database connection that no component uses.
- Some users supply their own metadata Secret and remove `kedaConnection`
because they use `authenticationRef`. For these users, the worker pod does not
start. Kubernetes shows `CreateContainerConfigError`.
I did a test on a local kind cluster with KEDA and this branch. I used
PgBouncer with `usePgbouncer: false` and an external `TriggerAuthentication`.
KEDA scaled the worker from 0 to 1 correctly. The worker pod had `KEDA_DB_CONN`
from `rel-metadata/kedaConnection`. `helm template` shows the same result for
MySQL.
The base branch also adds `KEDA_DB_CONN` to the worker. Thus, this is not a
regression. But the PR description says that the worker environment does not
get the connection. That is not correct.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
