This is an automated email from the ASF dual-hosted git repository. potiuk pushed a commit to branch backport-ci-sync-v3-4-test in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 22ae84dd1df658f45a7faf6f7c110e1adb1dd9c5 Author: Jarek Potiuk <[email protected]> AuthorDate: Sun Oct 25 19:38:31 2026 +0100 Reduce K8s test resource usage with single-node Kind clusters (#73998) * Reduce K8s test resource usage with single-node Kind clusters * Add test for forwarded node port matching Kind config and NodePort service * Refactor Kubernetes configuration and tests to use centralized port rendering (cherry picked from commit a5e5e1a74be25f18629a346f85028fde27439b66) Generated-by: Claude Opus 5 --- .github/workflows/kustomize-overlays-tests.yml | 6 +- contributing-docs/testing/k8s_tests.rst | 9 +-- dev/breeze/doc/05_test_commands.rst | 2 +- .../src/airflow_breeze/utils/kubernetes_utils.py | 34 +++++++-- dev/breeze/tests/test_kubernetes_utils.py | 86 ++++++++++++++++++++++ scripts/ci/kubernetes/kind-cluster-conf.yaml | 5 +- 6 files changed, 123 insertions(+), 19 deletions(-) diff --git a/.github/workflows/kustomize-overlays-tests.yml b/.github/workflows/kustomize-overlays-tests.yml index e51def2487c..5fef9c1ac8a 100644 --- a/.github/workflows/kustomize-overlays-tests.yml +++ b/.github/workflows/kustomize-overlays-tests.yml @@ -102,13 +102,13 @@ jobs: # convention (sources come from the base # image, not the workspace). # 5. upload-k8s-image -> loads the now-built kubernetes-variant - # image into every kind node, so kubelet + # image into the kind node, so kubelet # can resolve the chart's image reference # without hitting ghcr (which is private). # 6. deploy-airflow # 7. smoke-test-overlay -> renders the overlay, auto-preloads the # overlay's own images into the kind - # nodes (no per-overlay images list + # node (no per-overlay images list # needed - discovery walks the rendered # manifest), applies, walks the # STATUS.yaml verify: block with @@ -124,7 +124,7 @@ jobs: run: breeze k8s configure-cluster - name: "Build kubernetes-variant image on top of the restored PROD image" run: breeze k8s build-k8s-image --no-copy-local-sources - - name: "Upload kubernetes-variant image to kind nodes" + - name: "Upload kubernetes-variant image to the kind node" run: breeze k8s upload-k8s-image - name: "Deploy Airflow chart" run: breeze k8s deploy-airflow --executor CeleryExecutor diff --git a/contributing-docs/testing/k8s_tests.rst b/contributing-docs/testing/k8s_tests.rst index f0b121bcee7..3a6afc11221 100644 --- a/contributing-docs/testing/k8s_tests.rst +++ b/contributing-docs/testing/k8s_tests.rst @@ -281,7 +281,6 @@ The typical session for tests with Kubernetes looks like follows: apiServerPort: 48366 nodes: - role: control-plane - - role: worker extraPortMappings: - containerPort: 30007 hostPort: 18150 @@ -290,12 +289,11 @@ The typical session for tests with Kubernetes looks like follows: Creating cluster "airflow-python-3.11-v1.24.2" ... â Ensuring node image (kindest/node:v1.24.2) đŧ - â Preparing nodes đĻ đĻ + â Preparing nodes đĻ â Writing configuration đ â Starting control-plane đšī¸ â Installing CNI đ â Installing StorageClass đž - â Joining worker nodes đ Set kubectl context to "kind-airflow-python-3.11-v1.24.2" You can now use your cluster with: @@ -377,13 +375,11 @@ The typical session for tests with Kubernetes looks like follows: coredns-6d4b75cb6d-rwp9d 1/1 Running 0 71s coredns-6d4b75cb6d-vqnrc 1/1 Running 0 71s etcd-airflow-python-3.11-v1.24.2-control-plane 1/1 Running 0 84s - kindnet-ckc8l 1/1 Running 0 69s kindnet-qqt8k 1/1 Running 0 71s kube-apiserver-airflow-python-3.11-v1.24.2-control-plane 1/1 Running 0 84s kube-controller-manager-airflow-python-3.11-v1.24.2-control-plane 1/1 Running 0 84s - kube-proxy-6g7hn 1/1 Running 0 69s kube-proxy-dwfvp 1/1 Running 0 71s - kube-scheduler-airflow-python-3.11-v1.24.2-control-plane 1/1 Running 0 84s + kube-scheduler-airflow-python-3.11-v1.24.2-control-plane 1/1 Running 0 84s KinD Cluster API server URL: http://localhost:48366 Connecting to localhost:18150. Num try: 1 @@ -446,7 +442,6 @@ The typical session for tests with Kubernetes looks like follows: Good version of helm installed: 3.9.2 in /Users/jarek/IdeaProjects/airflow/kubernetes-tests/.venv/bin Stable repo is already added Uploading Airflow image ghcr.io/apache/airflow/main/prod/python3.11-kubernetes to cluster airflow-python-3.11-v1.24.2 - Image: "ghcr.io/apache/airflow/main/prod/python3.11-kubernetes" with ID "sha256:fb6195f7c2c2ad97788a563a3fe9420bf3576c85575378d642cd7985aff97412" not yet present on node "airflow-python-3.11-v1.24.2-worker", loading... Image: "ghcr.io/apache/airflow/main/prod/python3.11-kubernetes" with ID "sha256:fb6195f7c2c2ad97788a563a3fe9420bf3576c85575378d642cd7985aff97412" not yet present on node "airflow-python-3.11-v1.24.2-control-plane", loading... NEXT STEP: You might now deploy Airflow by: diff --git a/dev/breeze/doc/05_test_commands.rst b/dev/breeze/doc/05_test_commands.rst index 980caf471f0..84b1edcb590 100644 --- a/dev/breeze/doc/05_test_commands.rst +++ b/dev/breeze/doc/05_test_commands.rst @@ -703,7 +703,7 @@ The runner is overlay-agnostic. For every overlay it: * renders the overlay and substitutes ``RELEASE-NAME`` / ``NAMESPACE``, * **auto-preloads every ``image:`` referenced by the rendered manifest** - into the kind nodes via ``docker pull`` (with retry on Docker Hub + into the kind node via ``docker pull`` (with retry on Docker Hub rate limits) + ``kind load docker-image``, so the test does not flake on registry availability, * applies the overlay, diff --git a/dev/breeze/src/airflow_breeze/utils/kubernetes_utils.py b/dev/breeze/src/airflow_breeze/utils/kubernetes_utils.py index 9b9eb299bdb..d6e9c343b9b 100644 --- a/dev/breeze/src/airflow_breeze/utils/kubernetes_utils.py +++ b/dev/breeze/src/airflow_breeze/utils/kubernetes_utils.py @@ -379,6 +379,20 @@ def _get_kind_cluster_config_content(python: str, kubernetes_version: str) -> di return yaml.safe_load(config_path.read_text()) +# Must match nodePort in scripts/ci/kubernetes/nodeport.yaml. +FORWARDED_NODE_PORT = 30007 + + +def _render_kind_cluster_config(forwarded_port_number: int, k8s_api_server_port: int) -> str: + return ( + (SCRIPTS_CI_KUBERNETES_PATH / "kind-cluster-conf.yaml") + .read_text() + .replace("{{FORWARDED_NODE_PORT}}", str(FORWARDED_NODE_PORT)) + .replace("{{FORWARDED_PORT_NUMBER}}", str(forwarded_port_number)) + .replace("{{API_SERVER_PORT}}", str(k8s_api_server_port)) + ) + + def set_random_cluster_ports(python: str, kubernetes_version: str, output: Output | None) -> None: """ Creates cluster config file and returns sockets keeping the ports bound. @@ -390,11 +404,8 @@ def set_random_cluster_ports(python: str, kubernetes_version: str, output: Outpu f"[info]Random ports: K8S API: {k8s_api_server_port}, API Server: {forwarded_port_number}" ) cluster_conf_path = get_kind_cluster_config_path(python=python, kubernetes_version=kubernetes_version) - config = ( - (AIRFLOW_ROOT_PATH / "scripts" / "ci" / "kubernetes" / "kind-cluster-conf.yaml") - .read_text() - .replace("{{FORWARDED_PORT_NUMBER}}", str(forwarded_port_number)) - .replace("{{API_SERVER_PORT}}", str(k8s_api_server_port)) + config = _render_kind_cluster_config( + forwarded_port_number=forwarded_port_number, k8s_api_server_port=k8s_api_server_port ) cluster_conf_path.write_text(config) get_console(output=output).print(f"[info]Config created in {cluster_conf_path}:\n") @@ -402,12 +413,23 @@ def set_random_cluster_ports(python: str, kubernetes_version: str, output: Outpu get_console(output=output).print("\n") +def _extract_forwarded_host_port(conf: dict[str, Any]) -> int: + for node in conf["nodes"]: + for port_mapping in node.get("extraPortMappings", []): + if port_mapping["containerPort"] == FORWARDED_NODE_PORT: + return port_mapping["hostPort"] + raise ValueError( + f"No node in the kind cluster config has an extraPortMappings entry for " + f"containerPort {FORWARDED_NODE_PORT}. Delete and recreate the cluster." + ) + + def get_kubernetes_port_numbers(python: str, kubernetes_version: str) -> tuple[int, int]: conf = _get_kind_cluster_config_content(python=python, kubernetes_version=kubernetes_version) if not conf: return 0, 0 k8s_api_server_port = conf["networking"]["apiServerPort"] - api_server_port = conf["nodes"][1]["extraPortMappings"][0]["hostPort"] + api_server_port = _extract_forwarded_host_port(conf) return k8s_api_server_port, api_server_port diff --git a/dev/breeze/tests/test_kubernetes_utils.py b/dev/breeze/tests/test_kubernetes_utils.py new file mode 100644 index 00000000000..88dee2d715f --- /dev/null +++ b/dev/breeze/tests/test_kubernetes_utils.py @@ -0,0 +1,86 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +from __future__ import annotations + +from unittest.mock import patch + +import pytest +import yaml + +from airflow_breeze.utils.kubernetes_utils import ( + FORWARDED_NODE_PORT, + SCRIPTS_CI_KUBERNETES_PATH, + _render_kind_cluster_config, + get_kubernetes_port_numbers, +) + +# Rendered config of a cluster created before the switch to single-node clusters â +# port extraction must keep working for clusters that already exist on disk. +LEGACY_TWO_NODE_CONFIG = """\ +kind: Cluster +apiVersion: kind.x-k8s.io/v1alpha4 +networking: + ipFamily: ipv4 + apiServerAddress: "127.0.0.1" + apiServerPort: 48366 +nodes: + - role: control-plane + - role: worker + extraPortMappings: + - containerPort: 30007 + hostPort: 18150 + listenAddress: "127.0.0.1" + protocol: TCP +""" + +NO_FORWARDED_PORT_CONFIG = """\ +kind: Cluster +apiVersion: kind.x-k8s.io/v1alpha4 +networking: + ipFamily: ipv4 + apiServerAddress: "127.0.0.1" + apiServerPort: 48366 +nodes: + - role: control-plane +""" + + [email protected]( + "config", + [ + _render_kind_cluster_config(forwarded_port_number=18150, k8s_api_server_port=48366), + LEGACY_TWO_NODE_CONFIG, + ], + ids=["current-template", "legacy-two-node"], +) +@patch("airflow_breeze.utils.kubernetes_utils._get_kind_cluster_config_content", autospec=True) +def test_get_kubernetes_port_numbers_extracts_ports_from_any_node(mock_get_content, config): + mock_get_content.return_value = yaml.safe_load(config) + assert get_kubernetes_port_numbers(python="3.10", kubernetes_version="v1.30.13") == (48366, 18150) + + +@patch("airflow_breeze.utils.kubernetes_utils._get_kind_cluster_config_content", autospec=True) +def test_get_kubernetes_port_numbers_raises_when_forwarded_port_mapping_is_missing(mock_get_content): + mock_get_content.return_value = yaml.safe_load(NO_FORWARDED_PORT_CONFIG) + with pytest.raises(ValueError, match="extraPortMappings"): + get_kubernetes_port_numbers(python="3.10", kubernetes_version="v1.30.13") + + +def test_nodeport_service_exposes_forwarded_node_port(): + node_port_service = yaml.safe_load((SCRIPTS_CI_KUBERNETES_PATH / "nodeport.yaml").read_text()) + assert [port["nodePort"] for port in node_port_service["spec"]["ports"]] == [FORWARDED_NODE_PORT] diff --git a/scripts/ci/kubernetes/kind-cluster-conf.yaml b/scripts/ci/kubernetes/kind-cluster-conf.yaml index 4e891f80ddf..d9e93e9646b 100644 --- a/scripts/ci/kubernetes/kind-cluster-conf.yaml +++ b/scripts/ci/kubernetes/kind-cluster-conf.yaml @@ -22,10 +22,11 @@ networking: apiServerAddress: "127.0.0.1" apiServerPort: {{API_SERVER_PORT}} nodes: + # A single node on purpose: kind drops the control-plane NoSchedule taint only for single-node + # clusters, so the Airflow and KubernetesPodOperator pods are scheduled here. - role: control-plane - - role: worker extraPortMappings: - - containerPort: 30007 + - containerPort: {{FORWARDED_NODE_PORT}} hostPort: {{FORWARDED_PORT_NUMBER}} listenAddress: "127.0.0.1" protocol: TCP
