This is an automated email from the ASF dual-hosted git repository.
Miretpl pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/airflow.git
The following commit(s) were added to refs/heads/main by this push:
new 9e6126a19c1 Allow mounting the Airflow config as a directory to avoid
the subPath mount race (#70180)
9e6126a19c1 is described below
commit 9e6126a19c1bb15958219156f205be6cf01867f4
Author: Diego Álvarez <[email protected]>
AuthorDate: Mon Nov 16 22:52:46 2026 +0100
Allow mounting the Airflow config as a directory to avoid the subPath mount
race (#70180)
* Allow mounting the Airflow config as a directory to avoid the subPath
mount race
The chart mounts airflow.cfg (and airflow_local_settings.py) into every
component and
the KubernetesExecutor pod template via a subPath bind-mount of the config
ConfigMap.
subPath + ConfigMap hits a known Kubernetes race: on node/volume lifecycle
events
(kubelet volume reconstruction, node autoscaling) the pre-computed subPath
source inode
goes stale and runc fails the mount at container init with a StartError and
no
application log. It is intermittent, cross-component, and
runtime-version-independent.
Add an opt-in mountConfigAsDir (default false, existing behaviour
unchanged). When true,
the config ConfigMap is mounted as a directory at {airflowHome}/config and
AIRFLOW_CONFIG
is set to {airflowHome}/config/airflow.cfg. A directory-projected ConfigMap
volume has no
pre-computed subPath bind source, so it is immune to the race.
airflow_local_settings.py
keeps its existing path.
Refs: #33788
Co-authored-by: diego15ag <[email protected]>
* Name newsfragment after PR number
* Fix docs spellcheck in mountConfigAsDir description
Sphinx spellcheck (chart docs) flagged "inode" and "runc". Reword
"stale-inode" to "stale-mount" and backtick ``runc`` so both are accepted,
keeping the description precise.
Co-authored-by: diego15ag <[email protected]>
* Format test with ruff
* Reuse airflow_config_dir helper for the AIRFLOW_CONFIG env var
The helper and the env var both derived {airflowHome}/config independently.
Make the helper return the unquoted path and quote at the two call sites, so
the config subdirectory is encoded once. Rendered output is unchanged.
* Describe mountConfigAsDir in the values docs instead of justifying it
Use the reviewer's wording for the values.yaml comment and the schema
description; the rationale lives in the PR description. Drop the
newsfragment as not needed.
* Address review on the chart config-mount tests
Cover every component that includes the config mount helper, not only the
core five, and branch on kind for the CronJob pod path. Express the mount
and env assertions as jmespath filters and drop the length assertion that
checked nothing. Split the default-mount test so the default case renders
with no values, and move the airflowLocalSettings case to its own test.
---------
Co-authored-by: diego15ag <[email protected]>
---
chart/templates/_helpers.yaml | 14 ++++
.../helm_tests/airflow_aux/test_airflow_common.py | 79 ++++++++++++++++++++++
chart/values.schema.json | 6 ++
chart/values.yaml | 4 ++
4 files changed, 103 insertions(+)
diff --git a/chart/templates/_helpers.yaml b/chart/templates/_helpers.yaml
index 1f0074b3c66..4ddbeecafcb 100644
--- a/chart/templates/_helpers.yaml
+++ b/chart/templates/_helpers.yaml
@@ -63,6 +63,10 @@ If release name contains chart name it will be used as a
full name.
# Hard Coded Airflow Envs
- name: AIRFLOW_HOME
value: {{ .Values.airflowHome }}
+ {{- if .Values.mountConfigAsDir }}
+ - name: AIRFLOW_CONFIG
+ value: {{ printf "%s/airflow.cfg" (include "airflow_config_dir" .) | quote
}}
+ {{- end }}
{{- if .Values.enableBuiltInSecretEnvVars.AIRFLOW__CORE__FERNET_KEY }}
- name: AIRFLOW__CORE__FERNET_KEY
valueFrom:
@@ -736,7 +740,16 @@ server_tls_key_file = /etc/pgbouncer/server.key
{{- printf "%s-config" (include "airflow.fullname" .) }}
{{- end }}
+{{- define "airflow_config_dir" -}}
+ {{- printf "%s/config" .Values.airflowHome }}
+{{- end }}
+
{{- define "airflow_config_mount" -}}
+{{- if .Values.mountConfigAsDir }}
+- name: config
+ mountPath: {{ include "airflow_config_dir" . | quote }}
+ readOnly: true
+{{- else }}
- name: config
mountPath: {{ template "airflow_config_path" . }}
subPath: airflow.cfg
@@ -748,6 +761,7 @@ server_tls_key_file = /etc/pgbouncer/server.key
readOnly: true
{{- end }}
{{- end }}
+{{- end }}
{{/* Helper for service account name generation */}}
{{- define "_serviceAccountNameGen" -}}
diff --git a/chart/tests/helm_tests/airflow_aux/test_airflow_common.py
b/chart/tests/helm_tests/airflow_aux/test_airflow_common.py
index 8c3418f7559..0b9560d4860 100644
--- a/chart/tests/helm_tests/airflow_aux/test_airflow_common.py
+++ b/chart/tests/helm_tests/airflow_aux/test_airflow_common.py
@@ -652,6 +652,85 @@ class TestAirflowCommon:
for doc in docs:
assert expected_mount in
jmespath.search("spec.template.spec.initContainers[0].volumeMounts", doc)
+ def test_mount_config_as_dir(self):
+ components = [
+ "templates/scheduler/scheduler-deployment.yaml",
+ "templates/workers/worker-deployment.yaml",
+ "templates/api-server/api-server-deployment.yaml",
+ "templates/triggerer/triggerer-deployment.yaml",
+ "templates/dag-processor/dag-processor-deployment.yaml",
+ "templates/flower/flower-deployment.yaml",
+ "templates/cleanup/cleanup-cronjob.yaml",
+ "templates/database-cleanup/database-cleanup-cronjob.yaml",
+ "templates/jobs/migrate-database-job.yaml",
+ "templates/jobs/create-user-job.yaml",
+ ]
+ docs = render_chart(
+ values={
+ "mountConfigAsDir": True,
+ "airflowLocalSettings": "# local settings",
+ "executor": "CeleryExecutor,KubernetesExecutor",
+ "flower": {"enabled": True},
+ "cleanup": {"enabled": True},
+ "databaseCleanup": {"enabled": True},
+ },
+ show_only=components,
+ )
+ dir_mount = {"name": "config", "mountPath": "/opt/airflow/config",
"readOnly": True}
+ for doc in docs:
+ pod = "spec.jobTemplate.spec.template.spec" if doc["kind"] ==
"CronJob" else "spec.template.spec"
+ assert
jmespath.search(f"{pod}.containers[0].volumeMounts[?name=='config' &&
!subPath]", doc) == [
+ dir_mount
+ ]
+ assert (
+ jmespath.search(
+ f"{pod}.containers[0].volumeMounts[?subPath=='airflow.cfg'
|| subPath=='airflow_local_settings.py']",
+ doc,
+ )
+ == []
+ )
+ assert
jmespath.search(f"{pod}.containers[0].env[?name=='AIRFLOW_CONFIG'].value", doc)
== [
+ "/opt/airflow/config/airflow.cfg"
+ ]
+
+ def test_config_mounted_via_subpath_by_default(self):
+ docs =
render_chart(show_only=["templates/scheduler/scheduler-deployment.yaml"])
+ mounts = "spec.template.spec.containers[0].volumeMounts"
+ assert jmespath.search(f"{mounts}[?subPath=='airflow.cfg']", docs[0])
== [
+ {
+ "name": "config",
+ "mountPath": "/opt/airflow/airflow.cfg",
+ "subPath": "airflow.cfg",
+ "readOnly": True,
+ }
+ ]
+ assert
jmespath.search(f"{mounts}[?subPath=='airflow_local_settings.py']", docs[0]) ==
[]
+ assert jmespath.search(f"{mounts}[?name=='config' && !subPath]",
docs[0]) == []
+ assert
jmespath.search("spec.template.spec.containers[0].env[?name=='AIRFLOW_CONFIG']",
docs[0]) == []
+
+ def test_local_settings_mounted_via_subpath_when_set(self):
+ docs = render_chart(
+ values={"airflowLocalSettings": "# local settings"},
+ show_only=["templates/scheduler/scheduler-deployment.yaml"],
+ )
+ mounts = "spec.template.spec.containers[0].volumeMounts"
+ assert jmespath.search(f"{mounts}[?subPath=='airflow.cfg']", docs[0])
== [
+ {
+ "name": "config",
+ "mountPath": "/opt/airflow/airflow.cfg",
+ "subPath": "airflow.cfg",
+ "readOnly": True,
+ }
+ ]
+ assert
jmespath.search(f"{mounts}[?subPath=='airflow_local_settings.py']", docs[0]) ==
[
+ {
+ "name": "config",
+ "mountPath": "/opt/airflow/config/airflow_local_settings.py",
+ "subPath": "airflow_local_settings.py",
+ "readOnly": True,
+ }
+ ]
+
def test_priority_class_name(self):
docs = render_chart(
values={
diff --git a/chart/values.schema.json b/chart/values.schema.json
index ad7fd9aebd6..244f11b7012 100644
--- a/chart/values.schema.json
+++ b/chart/values.schema.json
@@ -691,6 +691,12 @@
"x-docsSection": "Common",
"default": "See values.yaml"
},
+ "mountConfigAsDir": {
+ "description": "Mount the Airflow config ConfigMap as a directory
(at ``{airflowHome}/config``) and set accordingly, instead of mounting
``airflow.cfg`` via ``subPath``.",
+ "type": "boolean",
+ "x-docsSection": "Kubernetes",
+ "default": false
+ },
"rbac": {
"description": "Enable RBAC (default on most clusters these
days).",
"type": "object",
diff --git a/chart/values.yaml b/chart/values.yaml
index dab6066525f..a551f664c29 100644
--- a/chart/values.yaml
+++ b/chart/values.yaml
@@ -313,6 +313,10 @@ airflowConfigAnnotations: {}
# 'airflow_local_settings' file as a string (templated)
airflowLocalSettings: ~
+# Mount the Airflow config ConfigMap as a directory (at
``{airflowHome}/config``) and set
+# accordingly, instead of mounting ``airflow.cfg`` via ``subPath``.
+mountConfigAsDir: false
+
# Enable RBAC (default on most clusters these days)
rbac:
# Specifies whether RBAC resources should be created