This is an automated email from the ASF dual-hosted git repository. shahar1 pushed a commit to branch gha-arc-rebase in repository https://gitbox.apache.org/repos/asf/airflow-ci-infra.git
commit aec106759990dbfe529c1c8b4fe4cfb1a0cfd3df Author: Shahar Epstein <[email protected]> AuthorDate: Thu Sep 17 13:00:36 2026 +0300 Replace static node groups with Karpenter and ARC runner scale sets The previous design (cluster-autoscaler, six managed node groups, legacy ARC with cert-manager) could not scale from zero without ASG tag workarounds and had no spot-price-aware instance selection. The CodeBuild experiment measured ~$104 per full Tests (AMD) run against a ~$15k/yr CI budget. Karpenter with EKS Pod Identity and ARC's supported scale-set mode brings the same run to ~$12-13 with one NodePool per architecture and no cert-manager or EBS CSI dependency. The design rationale lives in docs/eks-runner-architecture.md and ADR 0001; the runner image now matches the host contract airflow's CI graph expects. --- Airflow_CI.png | Bin 80422 -> 0 bytes CONTEXT.md | 47 ++++ README.rst | 122 +++++------ ...ks-standard-mode-with-self-managed-karpenter.md | 40 ++++ docs/eks-runner-architecture.md | 244 +++++++++++++++++++++ helm/{ci-helmfile.yaml => ci-helmfile.yaml.gotmpl} | 15 +- .../templates/namespaces/{infra.yaml => arc.yaml} | 9 +- helm/cluster-config/templates/namespaces/ci.yaml | 26 --- helm/infra-helmfile.yaml | 42 ++-- .../Chart.yaml} | 9 +- .../templates/ec2nodeclass.yaml | 71 ++++++ .../templates/nodepool-amd-shared.yaml | 65 ++++++ .../templates/nodepool-amd.yaml | 63 ++++++ .../templates/nodepool-arm-shared.yaml | 65 ++++++ .../templates/nodepool-arm.yaml | 63 ++++++ .../values.yaml} | 30 +-- helm/values/cert-manager.yaml | 20 -- helm/values/gha-runner-scale-set-controller.yaml | 5 +- .../gha-runner-scale-sets/common.yaml.gotmpl | 137 ++++++++++-- helm/values/gha-runner-scale-sets/runners.yaml | 97 +++++--- .../{cluster-autoscaler.yaml => karpenter.yaml} | 28 +-- runner/Dockerfile | 88 +++++++- terraform/README.rst | 26 ++- terraform/eks/.terraform.lock.hcl | 145 ++++++------ terraform/eks/asg_tags.tf | 67 ------ terraform/eks/eks.tf | 233 ++++---------------- terraform/eks/iam.tf | 134 ----------- terraform/eks/{vpc.tf => karpenter.tf} | 38 ++-- terraform/eks/{vpc.tf => outputs.tf} | 51 ++--- terraform/eks/provider.tf | 13 +- terraform/eks/runner_pod_identity.tf | 113 ++++++++++ terraform/eks/state.tf | 16 +- terraform/eks/variables.tf | 48 +--- terraform/eks/version.tf | 5 +- terraform/eks/vpc.tf | 22 +- 35 files changed, 1376 insertions(+), 821 deletions(-) diff --git a/Airflow_CI.png b/Airflow_CI.png deleted file mode 100644 index 82e5043..0000000 Binary files a/Airflow_CI.png and /dev/null differ diff --git a/CONTEXT.md b/CONTEXT.md new file mode 100644 index 0000000..901412a --- /dev/null +++ b/CONTEXT.md @@ -0,0 +1,47 @@ +# Airflow CI Infrastructure + +Self-hosted GitHub Actions runner infrastructure (EKS + ARC + Karpenter) that offloads +selected Apache Airflow CI workloads from GitHub-hosted runners to AWS. + +## Language + +### Workload categories + +**Trusted run**: +A workflow run whose code ref is not attacker-controllable: `schedule`, `push`, or +`workflow_dispatch` events on `apache/airflow` itself. The only category eligible for +self-hosted runners in phase 1. +_Avoid_: using "trusted" for committer-authored PRs — those are Committer runs. + +**Canary run**: +A full-matrix CI run of main, triggered by cron (`schedule`) or by a push to main. +"Scheduled canary" and "push canary" distinguish the trigger when it matters. + +**Committer run**: +A CI run of a PR authored by someone with merge access (regardless of whether it comes +from a fork or a branch). Not a Trusted run — the PR code is still unreviewed; serving +these from self-hosted runners is a separate, unresolved decision. +_Avoid_: trusted PR, member run. + +**Fork PR run**: +A CI run of a PR from a contributor without merge access. Never eligible for +self-hosted runners. + +### Infrastructure + +**Runner scale set**: +An ARC unit of runner capacity targeted by a single `runs-on` label (e.g. +`airflow-runner-amd-xl`). One helm release per scale set. +_Avoid_: runner deployment (legacy ARC term), runner pool. + +**Runner class**: +The size tier of a scale set, expressed as pod resource requests (CPU/memory), not as a +node type. Karpenter chooses the actual instance. + +**System node group**: +The small always-on EKS managed node group hosting the controllers (ARC, Karpenter, +listeners). Everything else runs on Karpenter-provisioned nodes. + +**Stash**: +The S3-backed mechanism for passing artifacts between jobs of a run (drop-in for the +`apache/infrastructure-actions/stash` actions). diff --git a/README.rst b/README.rst index 729c657..f3c1e43 100644 --- a/README.rst +++ b/README.rst @@ -53,7 +53,7 @@ you need to follow the steps below: 3. Update kubeconfig to access the EKS cluster:: - AWS_PROFILE=airflow aws eks --region us-east-2 update-kubeconfig --name airflow --alias airflow + AWS_PROFILE=airflow aws eks --region us-east-2 update-kubeconfig --name airflow-ci --alias airflow-ci This command will update the kubeconfig file located at `~/.kube/config` to include the EKS cluster. You can verify that the cluster is added by running `kubectl get nodes`. @@ -65,82 +65,82 @@ you need to follow the steps below: Airflow CI infrastructure architecture -------------------------------------- -.. image:: Airflow_CI.png - :width: 100% - :align: center - :alt: Airflow CI infrastructure architecture -Manage the infrastructure resources -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +The design is documented in `docs/eks-runner-architecture.md <docs/eks-runner-architecture.md>`_. +In short: an EKS cluster (standard mode) with a small always-on system node group runs +`Actions Runner Controller <https://github.com/actions/actions-runner-controller>`_ (runner +scale sets mode) and `Karpenter <https://karpenter.sh>`_; runner pods are ephemeral +docker-in-docker pods, one job per pod, on spot nodes that Karpenter provisions on demand +and reclaims when idle. -We use Terraform to manage the EKS cluster and other AWS resources, and Helm (with helmfile) to manage the -Kubernetes resources inside the cluster. +Deploying from scratch +~~~~~~~~~~~~~~~~~~~~~~ -For Terraform, you can find the detailed documentation here: `Terraform Documentation <terraform/README.rst>`_ +We use Terraform for the AWS resources and Helm (with helmfile) for everything inside the +cluster. The order matters on a fresh account: -And for Helm, you mainly need two commands: +1. **Terraform** — see `Terraform Documentation <terraform/README.rst>`_:: -.. code-block:: bash + cd terraform/eks + terraform init + terraform plan -out=terraform.tfplan + terraform apply terraform.tfplan - # diff the infra helmfile to compare the current state with the desired state - helmfile -f helm/infra-helmfile.yaml diff +2. **kubeconfig**:: - # apply the helmfiles to deploy the desired state - helmfile -f helm/infra-helmfile.yaml apply + aws eks --region us-east-2 update-kubeconfig --name airflow-ci --alias airflow-ci - # same for ci helmfile - helmfile -f helm/ci-helmfile.yaml diff - helmfile -f helm/ci-helmfile.yaml apply +3. **Infra helmfile** (namespaces, Karpenter, NodePools):: -For Docker, we build a custom image for the runners, based on the official `Dockerfiles`_ provided by Github, but -with the `Airflow runner release`_ managed by `Ash`_ instead of the official one, and with python installed on the image -(check `setup-python issue`_ for more details). + helmfile -f helm/infra-helmfile.yaml diff + helmfile -f helm/infra-helmfile.yaml apply -To build docker image, you can run the following command: +4. **GitHub App secret** — create a GitHub App (for repository-level runners: + ``Administration: Read & write`` + ``Metadata: Read-only``; for organization-level + runners: ``Self-hosted runners: Read & write``), install it on the target + repo/org, then:: -.. code-block:: bash + kubectl create secret generic github-config --namespace arc-runners \ + --from-literal=github_app_id=<app-id> \ + --from-literal=github_app_installation_id=<installation-id> \ + --from-file=github_app_private_key=<path-to-private-key.pem> - export TARGETPLATFORM=linux/arm64 - export AIRFLOW_RUNNER_VERSION=2.304.0-airflow8 - docker build --platform $TARGETPLATFORM \ - --build-arg AIRFLOW_RUNNER_VERSION=$AIRFLOW_RUNNER_VERSION \ - -t public.ecr.aws/u9s5q9f7/airflow-gha-runner:$AIRFLOW_RUNNER_VERSION \ - runner --push +5. **CI helmfile** (ARC controller + runner scale sets):: -.. _Dockerfiles: https://github.com/actions/actions-runner-controller/tree/master/runner + helmfile -f helm/ci-helmfile.yaml.gotmpl diff + helmfile -f helm/ci-helmfile.yaml.gotmpl apply -.. _Airflow runner release: https://github.com/ashb/runner/tree/v2.304.0-airflow8 +The runner image extends the official ``ghcr.io/actions/actions-runner`` image with the +host tools airflow's CI expects (sudo, docker CLI + compose, python3, ...). Images are +**date-tagged, never ``:latest``** — the tag referenced in +``helm/values/gha-runner-scale-sets/runners.yaml`` must always be an explicit, reviewed +pin. Build and push:: -.. _Ash: https://github.com/ashb - -.. _setup-python issue: https://github.com/actions/setup-python/issues/705 + TAG=$(date +%Y%m%d) + docker build --platform linux/amd64 \ + -t ghcr.io/apache/airflow-ci-infra/actions-runner:${TAG} runner + docker push ghcr.io/apache/airflow-ci-infra/actions-runner:${TAG} + # then update the image tag in helm/values/gha-runner-scale-sets/runners.yaml Some important notes ~~~~~~~~~~~~~~~~~~~~ -There are some configuration parameters that you need to be aware of: - -- EKS node group instance type: - In our current architecture, we use different types of runners, and each set of runners is configured via a - RunnerDeployment. Instead of setting container resources, we define the node selectors to schedule the - runner pod on a specific node group, and we add an anti-affinity to ensure the pod uses all the node resources - without sharing them with other runners. In this case, we need to carefully choose the instance types because they - represent the runners' resources. - -- ARC sync period: - The ARC sync period is the duration in which the controller reconciles the desired runners count. Increasing - this period could lead to a delay in the runners count update (more time in the queue when needing a scale-up - and a waste of resources when needing a scale-down). Decreasing it could lead to a high load on the Github API. - -- Autoscaler conf: - Similar to the ARC sync period, the autoscaler `scan-interval` is the duration in which the autoscaler checks the - cluster resources and scales up or down the cluster. Additionally, two other important parameters are - `scale-down-unneeded-time` and `scale-down-delay-after-add`, which control the delay before taking the scale-down - decision when the node is underutilized and the delay of checking the scaling-down necessity after a scale-up - decision. - -- HorizontalRunnerAutoscaler scaleDownDelaySecondsAfterScaleOut: - This parameter represents the delay before the HorizontalRunnerAutoscaler (HRA) takes the scale-down decision after - adding a new runner. Currently, we use a small delay (30 seconds) because the K8S autoscaler waits for 5 minutes - before deleting the node. So, if we need a runner, the HRA will add a new one, and it will be scheduled in less - than 20 seconds. +- Runner sizing is done with pod **resource requests** (see + ``helm/values/gha-runner-scale-sets/runners.yaml``), not node selectors/anti-affinity: + Karpenter picks the cheapest spot instance that fits the request. The current single + ``airflow-runner-amd-xl`` class requests a whole 32-vCPU node per job. +- Scale-up path: job queued → ARC listener requests a runner pod → pod Pending → + Karpenter launches a node (~60-90s). Scale-down: pod finishes → node empty → + Karpenter consolidates it after 60s (``WhenEmpty`` — running jobs are never evicted). +- The dockerd inside each runner is configured (dind sidecar args in + ``helm/values/gha-runner-scale-sets/common.yaml.gotmpl``) with a registry mirror, + widened address pools and nofile ulimits — these fix real failures observed with + airflow's CI and should not be removed casually. +- Workflows target the runners with ``runs-on: <scale-set-name>``, e.g. + ``runs-on: airflow-runner-amd-xl``. +- **Spot policy (phase 1): spot-first for everything, deliberately.** A spot reclaim + kills the job mid-run — the job shows red with a "runner lost communication" + annotation; just re-run the failed jobs. Tripwire: if interruptions exceed ~2/week + (or kill the same canary twice in a row), pin the heavy runner class to on-demand + (``karpenter.sh/capacity-type`` requirement in the NodePool) once size classes are + split. diff --git a/docs/adr/0001-eks-standard-mode-with-self-managed-karpenter.md b/docs/adr/0001-eks-standard-mode-with-self-managed-karpenter.md new file mode 100644 index 0000000..28f7b23 --- /dev/null +++ b/docs/adr/0001-eks-standard-mode-with-self-managed-karpenter.md @@ -0,0 +1,40 @@ +<!-- + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. +--> + +# EKS standard mode with self-managed Karpenter, not EKS Auto Mode + +Status: accepted (2026-07-25) + +The CI runner cluster runs EKS in standard mode with Karpenter installed and +managed by us, even though EKS Auto Mode is AWS's recommended default and would +remove Karpenter, AMI, and addon management entirely. Auto Mode nodes are +locked-down Bottlerocket appliances — SELinux enforcing, immutable read-only +root, no SSH/SSM, no node-level customization — and our runner pods are +privileged docker-in-docker workloads that run KinD (Kubernetes *inside* that +dind) and require node sysctls (`fs.inotify.*`, `nf_conntrack`, `fs.nr_open`); +this is precisely the workload class that fights a locked host, and it is +battle-tested on plain AL2023 with our own user data. Auto Mode also adds a +per-instance management fee (~10–12% of the instance price) to a +compute-dominated, budget-capped bill. + +Revisit if the workload ever stops needing privileged dind (e.g. jobs move to +rootless/kaniko-style builds and KinD is replaced), or if Auto Mode gains +supported node-level sysctl/privileged-workload customization at a fee that +undercuts our measured operational cost of self-managing Karpenter +(~a few hours per quarter). diff --git a/docs/eks-runner-architecture.md b/docs/eks-runner-architecture.md new file mode 100644 index 0000000..c9a71d8 --- /dev/null +++ b/docs/eks-runner-architecture.md @@ -0,0 +1,244 @@ +<!-- + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. +--> + +# EKS-based GitHub Actions runners for Airflow CI — architecture + +*Status: proposal (2026-07). Supersedes the 2023/2024 design on this branch (PR #59). +All versions and prices below were checked against official sources on 2026-07-25.* + +## Why this exists + +Airflow's canary/scheduled CI competes for the shared ASF GitHub Actions concurrency +pool. An AWS CodeBuild-managed-runner experiment (June–July 2026) proved the mechanics +work but the economics don't: a full "Tests (AMD)" run costs **≈$104** on CodeBuild +(1,300 build-minutes on 36 vCPU machines at $0.0798/min), ≈$135k+/yr for scheduled +runs alone — far beyond the ~$15k/yr CI budget envelope. Measured per-job utilization +shows a **7.3× oversizing waste** (median job peaks at ~3 vCPU / 2.3 GiB on a +36 vCPU / 72 GiB box) on top of CodeBuild's ~6.6× per-vCPU price premium over EC2 spot. +EKS + spot brings the same run to **≈$7–13**. + +## Architecture at a glance + +```mermaid +flowchart LR + subgraph github["GitHub"] + actions["GitHub Actions service"] + workflow["apache/airflow workflows<br/>runs-on: airflow-runner-amd-xl"] + workflow -->|"queue job"| actions + end + + subgraph aws["AWS account · us-east-2"] + subgraph vpc["airflow-ci-vpc — public subnets, no NAT"] + subgraph eks["EKS cluster 'airflow-ci' v1.34 (standard mode)"] + subgraph system["System node group — 2× t4g.medium on-demand"] + controller["ARC controller<br/>(arc-systems)"] + listener["ARC listener<br/>per scale set"] + karpenter["Karpenter v1.11<br/>(kube-system)"] + end + subgraph arcrunners["arc-runners namespace"] + secret["Secret github-config<br/>(GitHub App creds)"] + runner["Ephemeral runner pod<br/>runner + privileged dind sidecar<br/>requests 28 vCPU / 56 Gi · 1 job"] + end + node["Spot node — cheapest of<br/>c/m/r 6th-7th gen ≥ 8xlarge<br/>AL2023 · 200 Gi gp3 · IMDS hop 1"] + end + end + sqs["SQS queue<br/>airflow-ci-karpenter<br/>(spot interruptions)"] + s3[("S3 stash bucket<br/>airflow-ci-runner-cache-us-east-2")] + end + + listener -->|"outbound long-poll<br/>(no inbound access)"| actions + listener -->|"job queued:<br/>request runner"| runner + secret -.-> listener + runner -.->|"Pending pod"| karpenter + karpenter -->|"launch / consolidate<br/>(WhenEmpty, 60s)"| node + runner ---|"scheduled on"| node + sqs --> karpenter + runner -->|"EKS Pod Identity<br/>(scoped S3 only)"| s3 +``` + +Terraform (`terraform/eks/`) creates everything under **AWS**; helmfile +(`helm/infra-helmfile.yaml`, `helm/ci-helmfile.yaml.gotmpl`) deploys everything inside +**EKS**; the `github-config` secret is created once by hand at bootstrap. + +- **Terraform** (this repo, `terraform/eks/`): VPC, EKS cluster, system node group, + Karpenter controller IAM (Pod Identity), node IAM role, spot-interruption SQS queue. +- **Helmfile** (this repo, `helm/`): Karpenter, ARC controller, runner scale sets, + Karpenter `NodePool`/`EC2NodeClass` CRs, namespaces. +- **Airflow repo**: `runs-on: airflow-runner-amd-xl` on the Tests (AMD) job graph + (experiment branch; later behind the existing `selective_checks` runner seam for + canary-only routing). + +## Key decisions (and what was rejected) + +### EKS standard mode + self-managed Karpenter — not EKS Auto Mode + +Auto Mode was seriously considered (it removes Karpenter/AMI/addon management) and +rejected for this workload: + +- Auto Mode nodes are locked-down Bottlerocket: SELinux enforcing, immutable root FS, + no SSH/SSM. CI runners are privileged docker-in-docker pods that run KinD + (Kubernetes-in-Docker-in-Docker) and need node sysctls (`fs.inotify.*`, + `nf_conntrack`, `fs.nr_open`) — exactly the class of workload that fights a locked + host. Standard AL2023 nodes with our own user data are the proven path. +- Auto Mode adds a per-instance management fee (~10–12% of the instance price) on a + budget-constrained, compute-dominated bill. +- We already need Helm/CRD management for ARC; Karpenter via the same helmfile is + marginal extra complexity. + +### Karpenter v1.11 — not cluster-autoscaler + 6 static node groups + +The previous design (this branch pre-rebase) used six per-size/arch managed node +groups, ASG label/taint tags for scale-from-zero (`asg_tags.tf`), and +cluster-autoscaler. Karpenter replaces all of it: + +- Spot-price-aware instance selection across families (the "instance diversity" + discussion in PR #59 becomes a one-line requirement list). +- Scale-from-zero without ASG tag gymnastics → `asg_tags.tf` deleted. +- Consolidation and native spot-interruption handling (SQS) built in. +- One `NodePool` instead of N node groups; an ARM pool later is ~10 lines. + +### ARC runner scale sets 0.14.x — still the supported approach in 2026 + +`gha-runner-scale-set` (Autoscaling Runner Scale Sets mode) remains GitHub's +maintained ARC mode (latest chart 0.14.2, 2026-05). Ephemeral runners, JIT +registration, no cert-manager/webhook dependency (that was legacy ARC). Auth via a +**GitHub App** (recommended over PAT): for repo-level runners it needs +*Administration: R/W* + *Metadata: R*; for org-level, *Self-hosted runners: R/W*. + +### Utilities: what's in, what's out + +| Utility | Verdict | Why | +|---|---|---| +| Karpenter | **In** | Node autoscaling; replaces CA + static node groups; spot handling. Cost: negligible (runs on system nodes). | +| ARC (scale sets) | **In** | The runner integration itself. | +| metrics-server | **Out** | ARC scales on the job queue, not resource metrics; no HPA anywhere. Add only if we later want `kubectl top`. | +| cert-manager | **Out** (removed from branch) | Only legacy-mode ARC needed it for admission webhooks. Scale-set mode doesn't. | +| EBS CSI driver | **Out** (removed from branch) | No PersistentVolumes: runner work dirs + dind storage are node-local ephemeral (200 GiB gp3 root via `EC2NodeClass`). | +| cluster-autoscaler | **Out** (replaced) | See Karpenter. | +| External Secrets Operator | **Out** | Exactly one static secret (GitHub App key). ESO adds a controller + still needs a backing store + IAM. `kubectl create secret` at bootstrap is simpler and rotated rarely. Revisit if secrets multiply. | +| AWS Load Balancer Controller | **Out** | Nothing is exposed; ARC listeners do outbound long-poll only. No Service/Ingress of type LB. | + +### Networking: public subnets, no NAT — kept deliberately + +Nodes launch in public subnets with public IPs (as in the original PR, which removed +the NAT gateway). A NAT gateway costs ~$395/yr + data processing; public IPv4 on +ephemeral nodes costs $0.005/h only while nodes exist (≈$40/yr at our duty cycle). +Security posture is acceptable because: security groups allow no inbound; runners +only make outbound calls (GitHub long-poll, image pulls); nodes use IMDSv2 with hop +limit 1, so **pods (i.e. CI jobs) cannot reach node IAM credentials**; and the +node role carries only EKS/ECR/CNI policies. + +### Compute: one runner class now, right-sizing later + +Measured mix of a full Tests (AMD) run (duration-weighted minutes by required vCPU +class): `2:787, 4:209, 8:179, 16:96, 32:22, 48:6`. The cost-optimal design is 2–3 +scale-set size classes, but that requires per-job `runs-on` routing in the Airflow +repo. To keep the Airflow diff minimal for the experiment, we start with a **single +class sized so no job regresses** vs the CodeBuild baseline (32+ vCPU nodes): + +- Scale set `airflow-runner-amd-xl`: requests **28 vCPU / 56 Gi** → Karpenter picks + the cheapest current-gen ≥8xlarge (c6a/c7a/m6a/m7a/r6a/r7a, 32 vCPU) spot node; + one runner pod per node (requests-based, no anti-affinity hack). +- Spot first with automatic on-demand fallback (`capacity-type: [spot, on-demand]`). + Consolidation is `WhenEmpty` only — running jobs are never evicted to repack. +- maxRunners 100 (measured burst peak: 77 concurrent on the reduced run, ~163 on a + full canary; raise when canary-scale traffic is routed here). + +Cost at measured volume: ~21.7 node-hours per full run × ~$0.55–0.60/h (8xlarge +spot) ≈ **$12–13/run** vs $104 on CodeBuild — before phase-2 right-sizing, which +takes it to ~$4–5. + +**ARM later: yes, worth it.** Graviton spot is 10–25% cheaper per vCPU and the ARM +canary exists. It's one extra `NodePool` (arch: arm64) + one scale set + an ARM +runner image tag. Deliberately out of scope until AMD is proven. + +### dind configuration carries the CodeBuild lessons + +The runner pods define an explicit dind sidecar (not the chart's default) so dockerd +starts with flags that fix real failures seen on CodeBuild: + +- `--registry-mirror=https://mirror.gcr.io` — Docker Hub anonymous pull limits on + shared cloud egress IPs (also covers in-container `testcontainers` pulls). +- `--default-address-pool base=172.16.0.0/14,size=24` — parallel breeze compose + networks exhaust Docker's default ~31 subnets. +- `--default-ulimit nofile=65536:1048576` — venv-heavy tests exhaust fds; soft cap + kept moderate to avoid the close()-loop spawn trap. +- cgroup-v2 delegation for KinD: `docker:dind`'s entrypoint handles this correctly + (the CodeBuild failure existed precisely because its dockerd was *not* dind). +- Node sysctls (`fs.inotify.max_user_watches/instances`, `nf_conntrack_tcp_be_liberal`, + `fs.nr_open`, `fs.file-max`) are set in `EC2NodeClass` user data. + +### Terraform layout: one root module, S3 backend + +Single root module at `terraform/eks/` (no premature module split — the AWS-side +footprint is ~5 resources plus two registry modules). Existing S3 backend +(`airflow-ci-tfstate` / DynamoDB `TerraformLocks`, us-east-2) is kept for the real +deployment; it still holds the **stale 2024 state** of the deleted cluster (the +`airflow-vpc` 10.1.0.0/16 VPC still exists in the account!) — reconcile before the +first real apply (import-or-destroy; see README). Local iteration uses a +`*_override.tf` local backend so `plan` runs without touching shared state. The +provider no longer hardcodes `profile = "airflow"` — use `AWS_PROFILE`/env, which +also works for a future GitHub Actions plan/apply workflow via OIDC role assumption. + +Versions: Terraform ≥ 1.9, AWS provider ~> 6.0, `terraform-aws-modules/eks` ~> 21.24 +(EKS 1.34, standard support to 2027-12), `terraform-aws-modules/vpc` ~> 6.6, +Karpenter 1.11.3, ARC chart 0.14.2. + +### Security & trust boundary + +- Runners serve **trusted refs only**: scheduled/canary runs and (later) explicitly + label-gated committer PRs. Fork PRs stay on GitHub-hosted runners. For the org + deployment, the scale sets sit in a dedicated runner group restricted to + apache/airflow with "public repositories: selected workflows" disabled. +- GitHub App over PAT: scoped, auditable, revocable; key lives only in the + `arc-runners` namespace secret. +- Ephemeral runners: one job per pod, pod deleted after the job — no state bleed. +- Jobs can't reach node IAM credentials (IMDS hop limit 1); the node role has no + access beyond EKS basics; the AWS account contains no Airflow production secrets. +- Cluster API endpoint is public + access entries (admin = cluster creator + + explicitly added PMC operators); no aws-auth ConfigMap. + +## Scaling behavior + +1. Job queued on GitHub → ARC listener (long-poll) sees it, requests a runner pod. +2. Pod is Pending (no capacity) → Karpenter launches the cheapest fitting spot + instance (~60–90 s to Ready including image pull). +3. Job runs in the ephemeral runner (dind sidecar provides Docker). +4. Pod completes and is deleted → node goes empty → Karpenter consolidates it away + after 60 s. Idle cluster = 2× t4g.medium + control plane ≈ **$1.6k/yr fixed**. +5. Spot interruption → SQS → Karpenter drains/replaces; the interrupted job fails + and is retried by the workflow's normal retry/re-run mechanisms (acceptable for + canary; pin the pool to on-demand if measured interruption rates hurt). + +## Costs (measured workload, official prices) + +| | CodeBuild (today) | EKS this design | EKS after right-sizing | +|---|---|---|---| +| Full Tests (AMD) run | $104 | ~$12–13 | ~$4–5 | +| Scheduled-only, 2 AMD + 2 ARM per day, /yr | ~$135k+ | ~$20k | ~$9–11k | +| Fixed (cluster + system nodes + misc) | — | ~$1.8–2.1k/yr | same | + +## What changes in the Airflow repo (experiment branch) + +Only the Tests (AMD) job graph: `runs-on` swapped from the CodeBuild label array to +`airflow-runner-amd-xl`, and the CodeBuild-specific host guards extended to cover +the ARC runner environment. Everything else (selective checks, breeze, tests) is +untouched. The long-term mechanism is the existing `selective_checks` runner seam +(static labels work with ARC, unlike CodeBuild's per-run label), flipped for canary +runs only. diff --git a/helm/ci-helmfile.yaml b/helm/ci-helmfile.yaml.gotmpl similarity index 76% rename from helm/ci-helmfile.yaml rename to helm/ci-helmfile.yaml.gotmpl index e1da5a0..c707c9f 100644 --- a/helm/ci-helmfile.yaml +++ b/helm/ci-helmfile.yaml.gotmpl @@ -20,19 +20,26 @@ environments: values: - ./values/gha-runner-scale-sets/runners.yaml +# helmfile v1 requires environments and releases in separate YAML documents. +--- releases: - name: actions-runner-controller chart: oci://ghcr.io/actions/actions-runner-controller-charts/gha-runner-scale-set-controller - namespace: ci - version: 0.9.3 + namespace: arc-systems + version: 0.14.2 values: - ./values/gha-runner-scale-set-controller.yaml {{- range $key, $value := .Values.runnerScaleSets }} - name: {{ $key }} chart: oci://ghcr.io/actions/actions-runner-controller-charts/gha-runner-scale-set - namespace: ci - version: 0.9.3 + namespace: arc-runners + version: 0.14.2 + needs: + - arc-systems/actions-runner-controller + # AutoscalingRunnerSet CRD ships with the controller chart above — skip + # upfront validation so a fresh cluster can install in one pass. + disableValidationOnInstall: true values: - ./values/gha-runner-scale-sets/common.yaml.gotmpl {{- end }} diff --git a/helm/cluster-config/templates/namespaces/infra.yaml b/helm/cluster-config/templates/namespaces/arc.yaml similarity index 85% rename from helm/cluster-config/templates/namespaces/infra.yaml rename to helm/cluster-config/templates/namespaces/arc.yaml index f6bd908..50450f3 100644 --- a/helm/cluster-config/templates/namespaces/infra.yaml +++ b/helm/cluster-config/templates/namespaces/arc.yaml @@ -18,9 +18,14 @@ */}} ################################################################################ -# infra namespace where we create the infrastructure resources +# arc-systems: ARC controller + listeners; arc-runners: runner pods + secrets ################################################################################ apiVersion: v1 kind: Namespace metadata: - name: infra + name: arc-systems +--- +apiVersion: v1 +kind: Namespace +metadata: + name: arc-runners diff --git a/helm/cluster-config/templates/namespaces/ci.yaml b/helm/cluster-config/templates/namespaces/ci.yaml deleted file mode 100644 index 123ddc1..0000000 --- a/helm/cluster-config/templates/namespaces/ci.yaml +++ /dev/null @@ -1,26 +0,0 @@ -{{/* - Licensed to the Apache Software Foundation (ASF) under one - or more contributor license agreements. See the NOTICE file - distributed with this work for additional information - regarding copyright ownership. The ASF licenses this file - to you under the Apache License, Version 2.0 (the - "License"); you may not use this file except in compliance - with the License. You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - - Unless required by applicable law or agreed to in writing, - software distributed under the License is distributed on an - "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY - KIND, either express or implied. See the License for the - specific language governing permissions and limitations - under the License. -*/}} - -################################################################################ -# ci namespace where we create the CI resources -################################################################################ -apiVersion: v1 -kind: Namespace -metadata: - name: ci diff --git a/helm/infra-helmfile.yaml b/helm/infra-helmfile.yaml index d8c49e5..100dd84 100644 --- a/helm/infra-helmfile.yaml +++ b/helm/infra-helmfile.yaml @@ -15,38 +15,26 @@ # specific language governing permissions and limitations # under the License. --- -repositories: - - name: jetstack - url: https://charts.jetstack.io - - name: autoscaler - url: https://kubernetes.github.io/autoscaler - - name: aws-ebs-csi-driver - url: https://kubernetes-sigs.github.io/aws-ebs-csi-driver - releases: - name: cluster-config chart: ./cluster-config namespace: kube-system - # https://github.com/kubernetes-sigs/aws-ebs-csi-driver/tree/master/charts/aws-ebs-csi-driver - - name: aws-ebs-csi-driver - chart: aws-ebs-csi-driver/aws-ebs-csi-driver + # https://karpenter.sh/docs/getting-started/ + # IAM/SQS for Karpenter comes from terraform/eks/karpenter.tf (Pod Identity). + - name: karpenter + chart: oci://public.ecr.aws/karpenter/karpenter namespace: kube-system + version: 1.11.3 values: - - ./values/ebs-csi-dirver.yaml - - # https://cert-manager.io/docs/installation/helm/ - - name: cert-manager - chart: jetstack/cert-manager - namespace: infra - version: v1.15.3 - values: - - ./values/cert-manager.yaml + - ./values/karpenter.yaml - # https://github.com/kubernetes/autoscaler/tree/master/charts/cluster-autoscaler - - name: autoscaler - chart: autoscaler/cluster-autoscaler - namespace: infra - version: 9.37.0 - values: - - ./values/cluster-autoscaler.yaml + # NodePool/EC2NodeClass CRs — depends on the Karpenter CRDs. + - name: karpenter-resources + chart: ./karpenter-resources + namespace: kube-system + needs: + - kube-system/karpenter + # On a fresh cluster the CRDs don't exist until the karpenter release above + # installs, so the upfront diff/validation of this release must be skipped. + disableValidationOnInstall: true diff --git a/helm/values/ebs-csi-dirver.yaml b/helm/karpenter-resources/Chart.yaml similarity index 83% rename from helm/values/ebs-csi-dirver.yaml rename to helm/karpenter-resources/Chart.yaml index 7f8f0d0..e33e60d 100644 --- a/helm/values/ebs-csi-dirver.yaml +++ b/helm/karpenter-resources/Chart.yaml @@ -15,7 +15,8 @@ # specific language governing permissions and limitations # under the License. --- -controller: - serviceAccount: - annotations: - eks.amazonaws.com/role-arn: arn:aws:iam::827901512104:role/ebs-csi-driver +apiVersion: v2 +name: karpenter-resources +description: Karpenter NodePools and EC2NodeClasses for the Airflow CI runners +type: application +version: 0.1.0 diff --git a/helm/karpenter-resources/templates/ec2nodeclass.yaml b/helm/karpenter-resources/templates/ec2nodeclass.yaml new file mode 100644 index 0000000..b5058dd --- /dev/null +++ b/helm/karpenter-resources/templates/ec2nodeclass.yaml @@ -0,0 +1,71 @@ +{{/* + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. +*/}} +apiVersion: karpenter.k8s.aws/v1 +kind: EC2NodeClass +metadata: + name: ci +spec: + amiSelectorTerms: + - alias: al2023@latest + role: {{ .Values.nodeIamRoleName }} + subnetSelectorTerms: + - tags: + karpenter.sh/discovery: {{ .Values.clusterName }} + securityGroupSelectorTerms: + - tags: + karpenter.sh/discovery: {{ .Values.clusterName }} + associatePublicIPAddress: true + metadataOptions: + httpTokens: required + # Hop limit 1 keeps node IAM credentials unreachable from the runner + # containers (CI jobs must not get AWS access via IMDS). + httpPutResponseHopLimit: 1 + blockDeviceMappings: + - deviceName: /dev/xvda + ebs: + volumeSize: {{ .Values.nodeVolumeSize }} + volumeType: gp3 + iops: 4000 + throughput: 250 + encrypted: true + deleteOnTermination: true + userData: | + MIME-Version: 1.0 + Content-Type: multipart/mixed; boundary="BOUNDARY" + + --BOUNDARY + Content-Type: text/x-shellscript; charset="us-ascii" + + #!/bin/bash + # Kernel limits for KinD clusters and fd-heavy test suites; GitHub-hosted + # runners ship these, stock AL2023 does not. + modprobe nf_conntrack || true + sysctl -w fs.inotify.max_user_watches=524288 + sysctl -w fs.inotify.max_user_instances=512 + sysctl -w net.netfilter.nf_conntrack_tcp_be_liberal=1 || true + sysctl -w fs.nr_open=2097152 + sysctl -w fs.file-max=2097152 + # Disable core dumps: AL2023's default systemd-coredump handler is + # kernel-global, so a SEGV inside any container dumps the whole process + # through the node's journald first — under test-suite disk load that + # takes >60s and broke task-sdk's test_exit_by_signal[segv] (GitHub's + # Ubuntu runners short-circuit via apport instead). CI nodes are + # ephemeral; nobody reads their dumps. + sysctl -w "kernel.core_pattern=|/bin/false" + --BOUNDARY-- diff --git a/helm/karpenter-resources/templates/nodepool-amd-shared.yaml b/helm/karpenter-resources/templates/nodepool-amd-shared.yaml new file mode 100644 index 0000000..b2f2eab --- /dev/null +++ b/helm/karpenter-resources/templates/nodepool-amd-shared.yaml @@ -0,0 +1,65 @@ +{{/* + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. +*/}} +{{/* + On-demand pool for the bin-packed small/medium runner classes. Shared nodes + concentrate many jobs, so a single spot reclaim kills them all at once and a + rerun re-packs the same jobs into the same vulnerable shape (measured + 2026-07-27/28: reclaim storms of 5-57 events wiped whole retry batches + repeatedly; the on-demand premium is ~$1.5-2 per run). The xl class stays on + the spot-first `ci-amd` pool: one job per node, so reclaims stay cheap, + independent retries. +*/}} +apiVersion: karpenter.sh/v1 +kind: NodePool +metadata: + name: ci-amd-shared +spec: + template: + metadata: + labels: + airflow-ci/pool: amd-shared + spec: + nodeClassRef: + group: karpenter.k8s.aws + kind: EC2NodeClass + name: ci + taints: + - key: airflow-ci/runner + effect: NoSchedule + requirements: + - key: kubernetes.io/arch + operator: In + values: ["amd64"] + - key: kubernetes.io/os + operator: In + values: ["linux"] + - key: karpenter.sh/capacity-type + operator: In + values: ["on-demand"] + - key: karpenter.k8s.aws/instance-family + operator: In + values: ["c6a", "c7a", "m6a", "m7a", "r6a", "r7a", "c6i", "c7i", "m6i", "m7i"] + - key: karpenter.k8s.aws/instance-cpu + operator: In + values: ["8", "16", "32", "48"] + disruption: + consolidationPolicy: WhenEmpty + consolidateAfter: 60s + limits: + cpu: {{ .Values.amdSharedPoolCpuLimit | quote }} diff --git a/helm/karpenter-resources/templates/nodepool-amd.yaml b/helm/karpenter-resources/templates/nodepool-amd.yaml new file mode 100644 index 0000000..b053dba --- /dev/null +++ b/helm/karpenter-resources/templates/nodepool-amd.yaml @@ -0,0 +1,63 @@ +{{/* + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. +*/}} +apiVersion: karpenter.sh/v1 +kind: NodePool +metadata: + name: ci-amd +spec: + template: + metadata: + labels: + airflow-ci/pool: amd + spec: + nodeClassRef: + group: karpenter.k8s.aws + kind: EC2NodeClass + name: ci + # Only runner pods (which tolerate this) land on these nodes, so they + # never pin cluster infra and can be reclaimed the moment a job ends. + taints: + - key: airflow-ci/runner + effect: NoSchedule + requirements: + - key: kubernetes.io/arch + operator: In + values: ["amd64"] + - key: kubernetes.io/os + operator: In + values: ["linux"] + # Spot first; Karpenter falls back to on-demand when spot is unavailable. + - key: karpenter.sh/capacity-type + operator: In + values: ["spot", "on-demand"] + - key: karpenter.k8s.aws/instance-family + operator: In + values: ["c6a", "c7a", "m6a", "m7a", "r6a", "r7a", "c6i", "c7i", "m6i", "m7i"] + # Small enough sizes for the small/medium runner classes to get + # right-sized (or bin-packed) nodes; xl pods still land on >= 32 vCPU + # because their 28-vCPU request cannot fit anything smaller. + - key: karpenter.k8s.aws/instance-cpu + operator: In + values: ["8", "16", "32", "48"] + disruption: + # Never repack running CI jobs; only reap empty nodes. + consolidationPolicy: WhenEmpty + consolidateAfter: 60s + limits: + cpu: {{ .Values.amdPoolCpuLimit | quote }} diff --git a/helm/karpenter-resources/templates/nodepool-arm-shared.yaml b/helm/karpenter-resources/templates/nodepool-arm-shared.yaml new file mode 100644 index 0000000..b1c402b --- /dev/null +++ b/helm/karpenter-resources/templates/nodepool-arm-shared.yaml @@ -0,0 +1,65 @@ +{{/* + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. +*/}} +{{/* + On-demand pool for the bin-packed small/medium runner classes. Shared nodes + concentrate many jobs, so a single spot reclaim kills them all at once and a + rerun re-packs the same jobs into the same vulnerable shape (measured + 2026-07-27/28: reclaim storms of 5-57 events wiped whole retry batches + repeatedly; the on-demand premium is ~$1.5-2 per run). The xl class stays on + the spot-first `ci-arm` pool: one job per node, so reclaims stay cheap, + independent retries. +*/}} +apiVersion: karpenter.sh/v1 +kind: NodePool +metadata: + name: ci-arm-shared +spec: + template: + metadata: + labels: + airflow-ci/pool: arm-shared + spec: + nodeClassRef: + group: karpenter.k8s.aws + kind: EC2NodeClass + name: ci + taints: + - key: airflow-ci/runner + effect: NoSchedule + requirements: + - key: kubernetes.io/arch + operator: In + values: ["arm64"] + - key: kubernetes.io/os + operator: In + values: ["linux"] + - key: karpenter.sh/capacity-type + operator: In + values: ["on-demand"] + - key: karpenter.k8s.aws/instance-family + operator: In + values: ["c6g", "c7g", "c8g", "m6g", "m7g", "m8g", "r6g", "r7g", "r8g"] + - key: karpenter.k8s.aws/instance-cpu + operator: In + values: ["8", "16", "32", "48"] + disruption: + consolidationPolicy: WhenEmpty + consolidateAfter: 60s + limits: + cpu: {{ .Values.armSharedPoolCpuLimit | quote }} diff --git a/helm/karpenter-resources/templates/nodepool-arm.yaml b/helm/karpenter-resources/templates/nodepool-arm.yaml new file mode 100644 index 0000000..2d6e763 --- /dev/null +++ b/helm/karpenter-resources/templates/nodepool-arm.yaml @@ -0,0 +1,63 @@ +{{/* + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. +*/}} +apiVersion: karpenter.sh/v1 +kind: NodePool +metadata: + name: ci-arm +spec: + template: + metadata: + labels: + airflow-ci/pool: arm + spec: + nodeClassRef: + group: karpenter.k8s.aws + kind: EC2NodeClass + name: ci + # Only runner pods (which tolerate this) land on these nodes, so they + # never pin cluster infra and can be reclaimed the moment a job ends. + taints: + - key: airflow-ci/runner + effect: NoSchedule + requirements: + - key: kubernetes.io/arch + operator: In + values: ["arm64"] + - key: kubernetes.io/os + operator: In + values: ["linux"] + # Spot first; Karpenter falls back to on-demand when spot is unavailable. + - key: karpenter.sh/capacity-type + operator: In + values: ["spot", "on-demand"] + - key: karpenter.k8s.aws/instance-family + operator: In + values: ["c6g", "c7g", "c8g", "m6g", "m7g", "m8g", "r6g", "r7g", "r8g"] + # Small enough sizes for the small/medium runner classes to get + # right-sized (or bin-packed) nodes; xl pods still land on >= 32 vCPU + # because their 28-vCPU request cannot fit anything smaller. + - key: karpenter.k8s.aws/instance-cpu + operator: In + values: ["8", "16", "32", "48"] + disruption: + # Never repack running CI jobs; only reap empty nodes. + consolidationPolicy: WhenEmpty + consolidateAfter: 60s + limits: + cpu: {{ .Values.armPoolCpuLimit | quote }} diff --git a/helm/values/cluster-autoscaler.yaml b/helm/karpenter-resources/values.yaml similarity index 51% copy from helm/values/cluster-autoscaler.yaml copy to helm/karpenter-resources/values.yaml index 2b4bf57..48c6b18 100644 --- a/helm/values/cluster-autoscaler.yaml +++ b/helm/karpenter-resources/values.yaml @@ -15,21 +15,21 @@ # specific language governing permissions and limitations # under the License. --- -cloudProvider: aws -awsRegion: us-east-2 +clusterName: airflow-ci -autoDiscovery: - clusterName: "airflow" +# Must match node_iam_role_name in terraform/eks/karpenter.tf. +nodeIamRoleName: airflow-ci-runner-node -rbac: - create: true - serviceAccount: - create: true - name: "cluster-autoscaler" - annotations: - eks.amazonaws.com/role-arn: arn:aws:iam::827901512104:role/eks-autoscaler-role +# Root volume for runner nodes: CI image + KinD images + parallel compose +# stacks need far more than the 20Gi AL2023 default (CodeBuild boxes had 256GB). +nodeVolumeSize: 200Gi -extraArgs: - scan-interval: 10s - scale-down-delay-after-add: 5m - scale-down-unneeded-time: 5m +# Upper bound on total simultaneously provisioned runner capacity +# (~100 concurrent 32-vCPU runner nodes). +amdPoolCpuLimit: "3200" +# On-demand shared pool (small/medium classes): sized for the measured +# full-matrix peak (~60 nodes averaging well under 48 vCPU each). +amdSharedPoolCpuLimit: "2000" +# ARM (Graviton) pools mirror the AMD split; experiment-sized like amd was. +armPoolCpuLimit: "3200" +armSharedPoolCpuLimit: "2000" diff --git a/helm/values/cert-manager.yaml b/helm/values/cert-manager.yaml deleted file mode 100644 index be548e5..0000000 --- a/helm/values/cert-manager.yaml +++ /dev/null @@ -1,20 +0,0 @@ -# Licensed to the Apache Software Foundation (ASF) under one -# or more contributor license agreements. See the NOTICE file -# distributed with this work for additional information -# regarding copyright ownership. The ASF licenses this file -# to you under the Apache License, Version 2.0 (the -# "License"); you may not use this file except in compliance -# with the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, -# software distributed under the License is distributed on an -# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -# KIND, either express or implied. See the License for the -# specific language governing permissions and limitations -# under the License. ---- -installCRDs: true -nodeSelector: - node-type: default diff --git a/helm/values/gha-runner-scale-set-controller.yaml b/helm/values/gha-runner-scale-set-controller.yaml index 63ab994..1ed8177 100644 --- a/helm/values/gha-runner-scale-set-controller.yaml +++ b/helm/values/gha-runner-scale-set-controller.yaml @@ -18,8 +18,7 @@ replicaCount: 1 labels: - type: ci app: gha-runner-scale-set-controller -nodeSelector: - node-type: default +# Watch all namespaces (scale sets live in arc-runners); no nodeSelector needed: +# runner nodes are tainted, so the controller lands on the system node group. diff --git a/helm/values/gha-runner-scale-sets/common.yaml.gotmpl b/helm/values/gha-runner-scale-sets/common.yaml.gotmpl index 9ed5238..1b23173 100644 --- a/helm/values/gha-runner-scale-sets/common.yaml.gotmpl +++ b/helm/values/gha-runner-scale-sets/common.yaml.gotmpl @@ -17,45 +17,134 @@ --- {{ $config := index .Values.runnerScaleSets .Release.Name }} -githubConfigUrl: https://github.com/apache +githubConfigUrl: {{ .Values.githubConfigUrl }} +# Pre-created secret in the arc-runners namespace holding the GitHub App +# credentials (github_app_id / github_app_installation_id / github_app_private_key). githubConfigSecret: github-config -runnerGroup: airflow-ci - minRunners: {{ $config.minRunners }} maxRunners: {{ $config.maxRunners }} -containerMode: - type: "dind" - +# The dind sidecar is spelled out (instead of `containerMode: dind`) so dockerd +# gets flags that fix real failures measured on the CodeBuild runners: +# --registry-mirror Docker Hub anonymous pull limits on shared cloud +# egress IPs (also covers in-container testcontainers +# pulls that cannot use host credentials). +# --default-address-pool parallel breeze compose stacks exhaust Docker's +# default ~31 network pools; /14 carved into /24s +# gives 1024 networks. Must not overlap the VPC CIDR +# (10.1.0.0/16). +# --default-ulimit nofile fd-heavy venv tests need >1024; the soft cap stays +# at 65536 because processes that close(0..soft) on +# spawn make a high soft limit a performance trap. template: + metadata: + annotations: + # Karpenter must never drain a node out from under a running CI job — + # not for drift (e.g. EC2NodeClass changes) and not for consolidation. + # Involuntary events (spot reclaim) are unaffected. + karpenter.sh/do-not-disrupt: "true" spec: + # CI jobs must not see this cluster's credentials: with the token mounted, + # kubernetes clients inside tests prefer in-cluster config over the KinD + # kubeconfig and hit the EKS API as the runner SA (403s broke the K8S + # system-test teardowns). EKS Pod Identity (S3 stash) uses its own + # separately-projected token and is unaffected. + automountServiceAccountToken: false + initContainers: + - name: init-dind-externals + image: {{ .Values.image }} + command: ["cp", "-r", "/home/runner/externals/.", "/home/runner/tmpDir/"] + volumeMounts: + - name: dind-externals + mountPath: /home/runner/tmpDir + - name: dind + image: docker:dind + args: + - dockerd + - --host=unix:///var/run/docker.sock + - --group=$(DOCKER_GROUP_GID) + - --registry-mirror=https://mirror.gcr.io + - --default-address-pool=base=172.16.0.0/14,size=24 + - --default-ulimit=nofile=65536:1048576 + # Classic image store (overlay2), matching GitHub-hosted runners. + # The containerd store (dind default) breaks `kind load docker-image`: + # docker save emits OCI manifests whose blobs ctr import --digests + # can't resolve ("content digest ... not found"). + - --feature=containerd-snapshotter=false + env: + - name: DOCKER_GROUP_GID + value: "123" + securityContext: + privileged: true + restartPolicy: Always + startupProbe: + exec: + command: ["docker", "info"] + periodSeconds: 5 + failureThreshold: 24 + volumeMounts: + - name: work + mountPath: /home/runner/_work + - name: dind-sock + mountPath: /var/run + - name: dind-externals + mountPath: /home/runner/externals + # /tmp is shared with the runner (see runner container) so that + # `docker run -v /tmp/...` host paths resolve — the daemon lives + # here, not in the runner container. + - name: shared-tmp + mountPath: /tmp containers: - name: runner image: {{ .Values.image }} - command: [ "/home/runner/run.sh" ] + command: ["/home/runner/run.sh"] + env: + - name: DOCKER_HOST + value: unix:///var/run/docker.sock + - name: RUNNER_WAIT_FOR_DOCKER_IN_SECONDS + value: "120" + # Lets airflow-repo CI scripts detect the self-hosted K8s runner + # (guards for host-specific steps like the /mnt docker relocation). + - name: AIRFLOW_SELF_HOSTED_RUNNER + value: "eks" + # Class-sized xdist worker cap; run_unit_tests.sh only defaults + # PARALLELISM when unset, so this wins over its self-hosted fallback. + - name: PARALLELISM + value: {{ $config.parallelism | quote }} + resources: + requests: + cpu: {{ $config.cpu | quote }} + memory: {{ $config.memory }} + volumeMounts: + - name: work + mountPath: /home/runner/_work + - name: dind-sock + mountPath: /var/run + # Shared with the dind sidecar: tests bind-mount pytest tmpdirs + # (docker run -v /tmp/...) and the daemon resolves those paths in + # ITS filesystem. On GitHub-hosted runners daemon and job share one + # host; this emptyDir recreates that contract for /tmp. + - name: shared-tmp + mountPath: /tmp + volumes: + - name: work + emptyDir: {} + - name: dind-sock + emptyDir: {} + - name: dind-externals + emptyDir: {} + - name: shared-tmp + emptyDir: {} nodeSelector: - node-type: gha-runners - size: {{ $config.size }} - arch: {{ $config.arch }} + airflow-ci/pool: {{ $config.pool }} tolerations: - - key: node-type - operator: Equal - value: gha-runners + - key: airflow-ci/runner + operator: Exists effect: NoSchedule - affinity: - podAntiAffinity: - requiredDuringSchedulingIgnoredDuringExecution: - - labelSelector: - matchExpressions: - - key: actions.github.com/scale-set-name - operator: In - values: - - {{ .Release.Name }} - topologyKey: kubernetes.io/hostname controllerServiceAccount: name: actions-runner-controller-gha-rs-controller - namespace: {{ .Release.Namespace }} + namespace: arc-systems diff --git a/helm/values/gha-runner-scale-sets/runners.yaml b/helm/values/gha-runner-scale-sets/runners.yaml index 3c475fe..4d86536 100644 --- a/helm/values/gha-runner-scale-sets/runners.yaml +++ b/helm/values/gha-runner-scale-sets/runners.yaml @@ -15,35 +15,70 @@ # specific language governing permissions and limitations # under the License. --- -image: ghcr.io/apache/airflow-ci-infra/actions-runner:20240902-rc1 +# Always a date-pinned tag, never :latest — an unreviewed image change must not +# be able to break every canary at once. Rebuild monthly or when ARC's minimum +# runner version forces it. +# Fork-experiment override (public package on the fork owner's ghcr); switch to +# ghcr.io/apache/airflow-ci-infra/actions-runner:<tag> for the production deploy. +image: ghcr.io/shahar1/airflow-ci-actions-runner:20260731.2 + +# Repo URL for the fork experiment; switch to https://github.com/apache +# (org-level, with a restricted runner group) for the production deployment. +githubConfigUrl: https://github.com/shahar1/airflow + +# Three size classes, assigned per job from the measured per-job peaks of the +# 2026-07-26 go/no-go runs (airflow repo files/eks-class-mapping.md): each job's +# runs-on picks the smallest class fitting peak x 1.3 headroom on both CPU and +# memory (Non-DB providers is memory-bound at 35 GiB, hence xl). The xl request +# fills a whole 32-vCPU node (one job = one node); small/medium pods bin-pack +# when Karpenter batches several pending jobs onto one larger spot node. +# parallelism feeds the PARALLELISM env (xdist worker cap in run_unit_tests.sh) +# so test parallelism matches the class instead of oversubscribing small nodes. runnerScaleSets: - arc-small-amd: - minRunners: 0 - maxRunners: 30 - size: small - arch: x64 - arc-medium-amd: - minRunners: 0 - maxRunners: 30 - size: medium - arch: x64 - arc-large-amd: - minRunners: 0 - maxRunners: 30 - size: large - arch: x64 - arc-small-arm: - minRunners: 0 - maxRunners: 30 - size: small - arch: arm64 - arc-medium-arm: - minRunners: 0 - maxRunners: 30 - size: medium - arch: arm64 - arc-large-arm: - minRunners: 0 - maxRunners: 30 - size: large - arch: arm64 + # small/medium bin-pack onto shared nodes -> on-demand pool (a spot reclaim + # of a shared node kills every job on it and reruns re-pack the same batch; + # see nodepool-amd-shared.yaml). + airflow-runner-amd-small: + minRunners: 0 + maxRunners: 60 + pool: amd-shared + cpu: "4" + memory: 16Gi + parallelism: "4" + airflow-runner-amd-medium: + minRunners: 0 + maxRunners: 80 + pool: amd-shared + cpu: "8" + memory: 32Gi + parallelism: "8" + airflow-runner-amd-xl: + minRunners: 0 + maxRunners: 100 + pool: amd + cpu: "28" + memory: 56Gi + parallelism: "16" + # ARM (Graviton) classes mirror the AMD layout; the image tag is a + # multi-arch manifest so the same reference serves both architectures. + airflow-runner-arm-small: + minRunners: 0 + maxRunners: 60 + pool: arm-shared + cpu: "4" + memory: 16Gi + parallelism: "4" + airflow-runner-arm-medium: + minRunners: 0 + maxRunners: 80 + pool: arm-shared + cpu: "8" + memory: 32Gi + parallelism: "8" + airflow-runner-arm-xl: + minRunners: 0 + maxRunners: 100 + pool: arm + cpu: "28" + memory: 56Gi + parallelism: "16" diff --git a/helm/values/cluster-autoscaler.yaml b/helm/values/karpenter.yaml similarity index 68% rename from helm/values/cluster-autoscaler.yaml rename to helm/values/karpenter.yaml index 2b4bf57..4c5780f 100644 --- a/helm/values/cluster-autoscaler.yaml +++ b/helm/values/karpenter.yaml @@ -15,21 +15,17 @@ # specific language governing permissions and limitations # under the License. --- -cloudProvider: aws -awsRegion: us-east-2 +settings: + clusterName: airflow-ci + # Must match queue_name in terraform/eks/karpenter.tf. + interruptionQueue: airflow-ci-karpenter -autoDiscovery: - clusterName: "airflow" +# IAM comes from the EKS Pod Identity association created by Terraform — +# no serviceAccount role annotation needed. -rbac: - create: true - serviceAccount: - create: true - name: "cluster-autoscaler" - annotations: - eks.amazonaws.com/role-arn: arn:aws:iam::827901512104:role/eks-autoscaler-role - -extraArgs: - scan-interval: 10s - scale-down-delay-after-add: 5m - scale-down-unneeded-time: 5m +# Fits the 2x t4g.medium system nodes. +controller: + resources: + requests: + cpu: 500m + memory: 768Mi diff --git a/runner/Dockerfile b/runner/Dockerfile index 683028a..6fc9126 100644 --- a/runner/Dockerfile +++ b/runner/Dockerfile @@ -19,17 +19,97 @@ FROM ghcr.io/actions/actions-runner:latest USER root +# Host-level tools the airflow CI workflows expect on a runner — the target +# contract is GitHub's hosted ubuntu-24.04 image, restricted to what the CI +# graph actually invokes (pip, uv-via-pip, aws, parallel, docker CLI). Jobs +# also run `sudo apt-get install ...` themselves (as on GitHub-hosted +# runners), so the runner user gets passwordless sudo instead of us baking in +# every package. RUN apt-get update \ && apt-get install -y --no-install-recommends \ - ca-certificates curl nodejs npm wget unzip vim git jq build-essential netcat subversion \ + sudo ca-certificates curl wget zip unzip zstd git jq parallel build-essential netcat-openbsd \ libssl-dev libffi-dev python3 python3-venv python3-dev python3-pip python-is-python3 \ - && npm install -g yarn \ && install -m 0755 -d /etc/apt/keyrings \ && curl -fsSL https://download.docker.com/linux/ubuntu/gpg -o /etc/apt/keyrings/docker.asc \ && chmod a+r /etc/apt/keyrings/docker.asc \ && echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/ubuntu \ $(. /etc/os-release && echo "$VERSION_CODENAME") stable" | tee /etc/apt/sources.list.d/docker.list > /dev/null \ && apt-get update \ - && apt-get install -y docker-compose-plugin + && apt-get install -y --no-install-recommends docker-ce-cli docker-compose-plugin docker-buildx-plugin \ + && rm -rf /var/lib/apt/lists/* \ + && echo "runner ALL=(ALL) NOPASSWD: ALL" > /etc/sudoers.d/runner \ + && chmod 0440 /etc/sudoers.d/runner -USER runner \ No newline at end of file +# The actions-runner base ships a malformed tz config (/etc/timezone = "/UTC", +# localtime -> zoneinfo//UTC) which breaks local-timezone detection with +# 'ZoneInfo keys may not be absolute paths' (seen in shared timezones tests). +# Normalize to what GitHub-hosted runners have. +RUN ln -snf /usr/share/zoneinfo/Etc/UTC /etc/localtime && echo "Etc/UTC" > /etc/timezone + +# Same mechanism GitHub's runner image uses to make system `pip install` work +# on Ubuntu 24.04 despite PEP 668 (actions/runner-images install-python.sh): +# a global pip.conf, not deleting the EXTERNALLY-MANAGED marker. Non-root pip +# installs fall through to ~/.local, hence the PATH addition. +RUN printf '[global]\nbreak-system-packages = true\n' > /etc/pip.conf + +# JDKs for the java-sdk prek hooks (sync-java-sdk-supervisor-schema, ktlint, +# spotless run gradle on the host; toolchains reference Java 11 and 17). +# GitHub-hosted runners ship the Temurin set — mirror the versions we use. +RUN apt-get update \ + && apt-get install -y --no-install-recommends openjdk-11-jdk-headless openjdk-17-jdk-headless \ + && rm -rf /var/lib/apt/lists/* \ + && ln -s "/usr/lib/jvm/java-17-openjdk-$(dpkg --print-architecture)" /usr/lib/jvm/java-17 +ENV JAVA_HOME=/usr/lib/jvm/java-17 + +# gh CLI (job-summary and finalize steps use `gh api`/`gh run view`; present +# on GitHub-hosted runners). Official apt repo per cli.github.com docs. +RUN curl -fsSL https://cli.github.com/packages/githubcli-archive-keyring.gpg -o /etc/apt/keyrings/githubcli-archive-keyring.gpg \ + && chmod a+r /etc/apt/keyrings/githubcli-archive-keyring.gpg \ + && echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/githubcli-archive-keyring.gpg] https://cli.github.com/packages stable main" > /etc/apt/sources.list.d/github-cli.list \ + && apt-get update \ + && apt-get install -y --no-install-recommends gh \ + && rm -rf /var/lib/apt/lists/* + +# kubectl (hosted runners ship it; the k8s workflows' cluster-state dump steps +# call it directly on the host). Pinned to the cluster's minor via pkgs.k8s.io. +RUN curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.34/deb/Release.key -o /tmp/k8s.key \ + && gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg /tmp/k8s.key \ + && chmod a+r /etc/apt/keyrings/kubernetes-apt-keyring.gpg \ + && echo "deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.34/deb/ /" > /etc/apt/sources.list.d/kubernetes.list \ + && apt-get update \ + && apt-get install -y --no-install-recommends kubectl \ + && rm -rf /var/lib/apt/lists/* /tmp/k8s.key + +# Real python3.10 on PATH (uv-managed build; hosted runners ship toolcache +# pythons, so this matches the contract). Provider-compat jobs check out old +# airflow branches whose prek configs pin python3.10 hooks; without a local +# interpreter prek downloads its own managed CPython, and that download is +# broken on arm64 (sysconfigdata missing build_time_vars, first ARM run +# 2026-07-31). A PATH python short-circuits the download on both arches. +RUN pip install uv \ + && UV_PYTHON_INSTALL_DIR=/opt/uv-python uv python install 3.10 \ + && ln -s /opt/uv-python/cpython-3.10.*/bin/python3.10 /usr/local/bin/python3.10 \ + && /usr/local/bin/python3.10 -c "import sysconfig; sysconfig.get_path('stdlib')" \ + && pip uninstall -y uv + +# AWS CLI v2 (stash actions use `aws s3`); official installer, arch-aware. +RUN ARCH=$(uname -m) \ + && curl -fsSL "https://awscli.amazonaws.com/awscli-exe-linux-${ARCH}.zip"; -o /tmp/awscliv2.zip \ + && unzip -q /tmp/awscliv2.zip -d /tmp \ + && /tmp/aws/install \ + && rm -rf /tmp/aws /tmp/awscliv2.zip + +ENV PATH="/home/runner/.local/bin:${PATH}" + +# Hosted runners export USER; workflows rely on it (e.g. `sudo chown -R +# "${USER}"` — with USER unset that chown is a silent no-op, exit 0, leaving +# root-owned dirs that later non-sudo steps can't write). +ENV USER=runner + +# breeze saves/loads CI image tars via `--image-file-dir /mnt` — on GitHub-hosted +# runners /mnt is the big scratch disk made writable by make_mnt_writeable.sh, +# which self-hosted runs deliberately skip. Backed here by the node's 200 GB +# root volume; sticky like /tmp. +RUN mkdir -p /mnt && chmod 1777 /mnt + +USER runner diff --git a/terraform/README.rst b/terraform/README.rst index ad8e00b..e9832c3 100644 --- a/terraform/README.rst +++ b/terraform/README.rst @@ -28,23 +28,37 @@ Requirements In order to deploy the infrastructure, you need to have the following tools installed: -- `tfenv <https://github.com/tfutils/tfenv>`_ to manage Terraform versions +- Terraform >= 1.9 (e.g. via `tfenv <https://github.com/tfutils/tfenv>`_: ``tfenv install 1.9.5``) - Configure your AWS and EKS credentials :ref:`doc <aws_configuration>` +State +~~~~~ -Configure Terraform -~~~~~~~~~~~~~~~~~~~ +The remote state lives in the ``airflow-ci-tfstate`` S3 bucket (``us-east-2``) with the +``TerraformLocks`` DynamoDB table. Credentials/profile are taken from the environment +(``AWS_PROFILE=...``), which also works for a future GitHub Actions plan/apply workflow +assuming a role via OIDC — nothing is hardcoded in the provider. -.. code-block:: bash +.. note:: - cd eks - tfenv install 1.5.3 + The bucket also contains ``tf-state/eks.tfstate`` — the **2024 deployment's** state. + Its cluster is gone but leftovers (the ``airflow-vpc`` VPC, old node-group IAM roles, + the ``/aws/eks/airflow/cluster`` log group, the ``alias/eks/airflow`` KMS alias) + still exist in the account and are **deliberately left untouched**. The current + deployment therefore uses its own state key (``tf-state/eks-airflow-ci.tfstate``) + and the cluster name ``airflow-ci``, so every derived resource name avoids the 2024 + ones. Do not import, modify, or delete the 2024 resources from this configuration. + For local iteration without touching the shared state, drop a + ``backend_override.tf`` (gitignored) with ``terraform { backend "local" {} }`` + next to the sources. Plan and deploy the infrastructure ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. code-block:: bash + cd eks + # Initialize the Terraform modules and install the required providers, # just the first time or when you add a new module AWS_PROFILE=airflow terraform init diff --git a/terraform/eks/.terraform.lock.hcl b/terraform/eks/.terraform.lock.hcl index a3a0c54..b5d5d74 100644 --- a/terraform/eks/.terraform.lock.hcl +++ b/terraform/eks/.terraform.lock.hcl @@ -2,104 +2,109 @@ # Manual edits may be lost in future updates. provider "registry.terraform.io/hashicorp/aws" { - version = "5.64.0" - constraints = ">= 4.33.0, >= 5.46.0, >= 5.61.0, ~> 5.64.0" + version = "6.56.0" + constraints = ">= 6.0.0, ~> 6.0, >= 6.28.0, >= 6.52.0" hashes = [ - "h1:YH4I78rsS9t+YoGMPNzrM53aWi0Rb9Nud16iusrSXMg=", - "zh:1d361f8062c68c9d5ac14b0aa8390709542129b8a9b258e61bbbabc706078b44", - "zh:39dcbf53e3896bdd77071384c8fad4a5862c222c73f3bcf356aca488101f22fd", - "zh:3fad63505f0c5b6f01cc9a6ef02b2226983b79424126a9caf6eb724f654299f4", - "zh:53a8b90d00829cc27e3171a13a8ff1404ee0ea018e73f31d3f916d246cc39613", - "zh:5734c25ef5a04b40f3c1ac5f817f11e42ee3328f74dbc141c0e64afbb0acc834", - "zh:66ea14dbd87f291ce4a877123363933d3ca4022f209f885807a6689c22c24e80", - "zh:68e79654ad0894a3d93134c3377748ace3058d5fad5ec09d1e9a8f8f9b8a47ea", - "zh:7b74259d0ceef0c49cea6bcd171df997b6bad141085bbadded15b440faeb0eee", - "zh:988ebfb5d115dc57070b5abf2e4200ad49cde535f27fd2ba5e34cf9ab336a57f", + "h1:iWz9BgFQaDPA1ChVGYvgI3MlUnx3wSQCA2ggYqDPcz8=", + "zh:2b3fbb3bebcc663b85d5fd9bbc2d131ab89322d696ff5c6ac6b7ffb7b5fe92e7", + "zh:30e56ccc7f33a7778ab323a28fe893d8e9200dc5fb92ccb7023bee808db3c1b0", + "zh:67dca271bef16547ef8ab5a6349f9bce39d91d7c1ae3d8388ada687ca774ba44", + "zh:824c812695b14d2fddad5e22339d4520e16d9c875f5d5095f29003f49a6fd124", + "zh:8372b12e30078d1df8b52e1285fd0d9d35160a7d18c3b0211f55229e5a832fd3", + "zh:8922b45ab65e272e8951f70d890444ddb3441170d8fa6f51298f24f20d21943d", + "zh:8fc4fb94ac8547717128cbcf296ac0ba0918738c4882029cbba46bd4812eb5e4", "zh:9b12af85486a96aedd8d7984b0ff811a4b42e3d88dad1a3fb4c0b580d04fa425", - "zh:a0a2d4efe2835f0101a0a5024e044a3f28c00e10a8d87fce89c707ef6db75cea", - "zh:aecb3e4b9121771dee9cac7975bf5d0657b5f3e8b57788c455beaeb0f3c48d93", - "zh:d2d3393170b8ef761d3146f39f6788c4a3e876e6c5d4cedca4870c2680688ae6", - "zh:daba5a005c1baa4a5eefbfb86d43ccf880eb5b42e8136f0d932f55886d72bda0", - "zh:de16a6ff3baacdaf9609a0a89aa1913fc19cccaf5ee0fc1c49c5a075baa47c02", + "zh:a2efcd0174b79c1dffce48a5984f137ebc6f67d2c2ee966b47210e1faeb05cf2", + "zh:a4a50aa269a19c1d664b0dd59ee8047ed8a4a5b449f54733518309feccce1f43", + "zh:a830d80316b3c3127c9e0c77b9d4f50702c9d1a884c08973ac50b326d9ac734a", + "zh:d7e1b9bb2df3cdde8381ab101a807ae54e72c156d13a6b73dae2b922dca0c9f5", + "zh:d87c2a1cfc03b01cf13dff3700898ab990e1817326728824da8499dd0129da72", + "zh:fbb1848a597263c66dc6587ec8c3e0586e505e36f99d23752763534f62a3fef7", + "zh:fcb54d08eb3c763f01223e12b4eacbd018478e8e67c6b8611940564dfbea3d90", + "zh:ff6df70b2b2f75bd9000630f131c02e6bc2b9172cdb2547030f3fc019a3f3bc5", ] } provider "registry.terraform.io/hashicorp/cloudinit" { - version = "2.3.4" + version = "2.4.0" constraints = ">= 2.0.0" hashes = [ - "h1:S3j8poSaLbaftlKq2STBkQEkZH253ZLaHhBHBifdpBQ=", - "zh:09f1f1e1d232da96fbf9513b0fb5263bc2fe9bee85697aa15d40bb93835efbeb", - "zh:381e74b90d7a038c3a8dcdcc2ce8c72d6b86da9f208a27f4b98cabe1a1032773", - "zh:398eb321949e28c4c5f7c52e9b1f922a10d0b2b073b7db04cb69318d24ffc5a9", - "zh:4a425679614a8f0fe440845828794e609b35af17db59134c4f9e56d61e979813", - "zh:4d955d8608ece4984c9f1dacda2a59fdb4ea6b0243872f049b388181aab8c80a", + "h1:Bx3XQkBSY3RAGwLZb8hyi8AhvahPNlt4mlyZhW9guOI=", + "zh:1b0fe71b8e87a068f7cd9faaa733100ab72ab61ce812b8bd2b8e3e6ea3907b2d", + "zh:2aa9631ad64cfda1eb58f147619b631dadedfaf9453b422aa5ada2d3861183c1", + "zh:2c5f35463bdfb2f87d3576b81e62c30f8109e67bb6f21ffcbc46a855811455c0", + "zh:5970bcad151ea236bd262ada1a5a23bfbc1716f94a4e8b16ab2bcdda91d6a671", "zh:78d5eefdd9e494defcb3c68d282b8f96630502cac21d1ea161f53cfe9bb483b3", - "zh:a48fbee1d58d55a1f4c92c2f38c83a37c8b2f2701ed1a3c926cefb0801fa446a", - "zh:b748fe6631b16a1dafd35a09377c3bffa89552af584cf95f47568b6cd31fc241", - "zh:d4b931f7a54603fa4692a2ec6e498b95464babd2be072bed5c7c2e140a280d99", - "zh:f1c9337fcfe3a7be39d179eb7986c22a979cfb2c587c05f1b3b83064f41785c5", - "zh:f58fc57edd1ee3250a28943cd84de3e4b744cdb52df0356a53403fc240240636", - "zh:f5f50de0923ff530b03e1bca0ac697534d61bb3e5fc7f60e13becb62229097a9", + "zh:79a0676909732b6ec0441a733af6383513cde3bd2cef5c1ad0a74131e1286a04", + "zh:818f16141481a1202b3977becd19a12d4d46cd2e3f5753f5d0d0049adacf8f8c", + "zh:948d98716831087e69eca99f91ed7964cc537f3aca279f7494645ee56c9dc4ec", + "zh:a75e78889565a51df3e8e3af207e36e5ddb25e47ce1780a784c82dc3c3109b67", + "zh:a9c6e455d52b1bba5272bd87a35cfabcfd6d903dcbe42e2de926228dbb1e39b2", + "zh:b846805d8c2f5d1d6c2ffeeaf32109d9af7db7fa3c56929bfc1dcfaadf9c8bd8", + "zh:c3e5279756b46c4f49a6f4c81347fbe2fffebb2bf18a5c24664830304a1f6a8e", + "zh:c8be7b31893163d0046b0137a6100533f07e8efd192a1903b6bb4c42be12dceb", ] } provider "registry.terraform.io/hashicorp/null" { - version = "3.2.2" + version = "3.3.0" constraints = ">= 3.0.0" hashes = [ - "h1:IMVAUHKoydFrlPrl9OzasDnw/8ntZFerCC9iXw1rXQY=", - "zh:3248aae6a2198f3ec8394218d05bd5e42be59f43a3a7c0b71c66ec0df08b69e7", - "zh:32b1aaa1c3013d33c245493f4a65465eab9436b454d250102729321a44c8ab9a", - "zh:38eff7e470acb48f66380a73a5c7cdd76cc9b9c9ba9a7249c7991488abe22fe3", - "zh:4c2f1faee67af104f5f9e711c4574ff4d298afaa8a420680b0cb55d7bbc65606", - "zh:544b33b757c0b954dbb87db83a5ad921edd61f02f1dc86c6186a5ea86465b546", - "zh:696cf785090e1e8cf1587499516b0494f47413b43cb99877ad97f5d0de3dc539", - "zh:6e301f34757b5d265ae44467d95306d61bef5e41930be1365f5a8dcf80f59452", + "h1:l+dm3lhmu4ys7GbvIldfn544olSPH0DOiYruuFSfQkY=", + "zh:021748b5ea3b5f6956f2e75c42c5cdc113b391fb98ac71364a4965d23b37000f", + "zh:3b27956f8541d46704fda234e0d535c2ae2a4b33411848b1ee262a1ec03568b0", + "zh:3de4ed47d6d0f4d8edba4a5092c7c9799950eda63989d8d0d2586e6afcb0aa20", + "zh:57ed8935c7d56dbc91cf2673534582cacfaab7a2f105f51d9f797e99df0c0c47", + "zh:58e176ba1d142827089e30e0711e007309a9f2726e8881986da5026e9778fdf4", + "zh:5949c4a3d4a93f841f155cdb7e991c087e637145c1630572e21948224f8f4923", + "zh:76d60f366b743003c1b085afa769b45b2198ee919927e45807d7d44fb42c067d", "zh:78d5eefdd9e494defcb3c68d282b8f96630502cac21d1ea161f53cfe9bb483b3", - "zh:913a929070c819e59e94bb37a2a253c228f83921136ff4a7aa1a178c7cce5422", - "zh:aa9015926cd152425dbf86d1abdbc74bfe0e1ba3d26b3db35051d7b9ca9f72ae", - "zh:bb04798b016e1e1d49bcc76d62c53b56c88c63d6f2dfe38821afef17c416a0e1", - "zh:c23084e1b23577de22603cff752e59128d83cfecc2e6819edadd8cf7a10af11e", + "zh:79cd1bab1261a07f84e917191d7ddc4340ac5f5524283767256f7ffd7f87caf0", + "zh:8ec9083038cf710b30e319eaa467c9df7fa52bbd9969b61053a35bc2cdd2e0a6", + "zh:a6e502cb579685ab7aeb886c2bb11ddd9cfed74b41008592d57cbc3351a9218b", + "zh:acb74d6b4f66ff6acfcda315df802a7432170ef3955c9b432cb4580767004006", + "zh:f0ce55d8d9ffdb33dab612b1246f9bab060a9d54fc32ce2b4a038646155660af", ] } provider "registry.terraform.io/hashicorp/time" { - version = "0.12.0" + version = "0.14.0" constraints = ">= 0.9.0" hashes = [ - "h1:Os2Ok7txtlUJHh6Hg7o+74Ql85SnRb/fGmah22yXpLw=", - "zh:019a4c09af254ef80b72cf0d843dfe72d99483e227138cf5b514a1b9977ab4c3", - "zh:0ae310ec740ebc6f275529507d60bb747d0bf39e72fc5a2fa90d74486006132c", - "zh:13d6aec117f05237fbf8c7d91d6ebb19797b00aa87e7a812642d3ea4738a394e", - "zh:2e87abbc261f9317d0c2ef26e01d5fabf77679da7d2cac6f47df7d198f720989", - "zh:4a6d471176ce0264455aa7d5457b8702f78400010c201c1719708958a1b7b647", + "h1:4EThC3ocCFiFPMZQSUvSGSxoJqBcGWxMcFYmL67uS7Y=", + "zh:12abfd6b800e4d7fa6db7310dec8ffd440b31993861ef188c7ed5260b3073937", + "zh:23005521e800bb19e1597bf755c5f70d675d30b685d4255001ed5fa47d9df3f1", + "zh:2fea249b582ae97cd1cc10385187ea50993bb47c28cc5df0305e57ceaabf0a10", + "zh:322018d3b987b7aad08697178029a2bb667bed699e88328f0c89c52a2fd41341", + "zh:32a08e98fce2d273cb9b2c89d6c54727cc9f0a32e15bfd896be4e02cc6b48f95", + "zh:3db89aabd0e619616bd4b0f8b373a7586dfe60feffcea12a84a0bdbc445714b3", + "zh:7488f56c81d742dc020f29063626c8f07ca188aa97be61e7307e8d62397020a2", "zh:78d5eefdd9e494defcb3c68d282b8f96630502cac21d1ea161f53cfe9bb483b3", - "zh:8978d5474a6da30bc0ad21c17db188d6918cacf3df3f6506b72ef3a268d53e2e", - "zh:b109efe138dfcb45dc04a9cc6809d185ab8b0ebc12040847c2dac430fda5af68", - "zh:b58e039b9106ac0a8de3c07f53b5279d7f0215fb35f2d23df642dfce0875382f", - "zh:ba2cbb2e515922d13efe3a46647be84f5426fcfcaa0f1520b3efeab8db847ed3", - "zh:c6c1ef1f26f25bca3abb5e07fa33dca37ed39cc26d0ff877964f2ffe5edd618c", - "zh:f8e171f923b7d2e789abd034072465dec3e6133c3a7644b7a7a965a74d52224e", + "zh:7cb4067f2e7559b13f7562ef722f948950901eb37834873e98360ab28f66e9d7", + "zh:9d552c8345f61e1b7db8e725144981345f18ac1014d58d6f5ddf0928a195fffb", + "zh:a8e69fb6b97fc9d86fb19a9f4d42abe33c4a68e700b15387ce2e17d2b9934bed", + "zh:aeeb900eb8dd0f790c60ea5c0e0c8d42bd6e4a54f391681d4decca15b544394b", + "zh:c239c619101a8c95e1f14061eb973c57a8d15fa0e68878ced5bbd76858ee5b79", ] } provider "registry.terraform.io/hashicorp/tls" { - version = "4.0.5" - constraints = ">= 3.0.0" + version = "4.3.0" + constraints = ">= 4.0.0" hashes = [ - "h1:zeG5RmggBZW/8JWIVrdaeSJa0OG62uFX5HY1eE8SjzY=", - "zh:01cfb11cb74654c003f6d4e32bbef8f5969ee2856394a96d127da4949c65153e", - "zh:0472ea1574026aa1e8ca82bb6df2c40cd0478e9336b7a8a64e652119a2fa4f32", - "zh:1a8ddba2b1550c5d02003ea5d6cdda2eef6870ece86c5619f33edd699c9dc14b", - "zh:1e3bb505c000adb12cdf60af5b08f0ed68bc3955b0d4d4a126db5ca4d429eb4a", - "zh:6636401b2463c25e03e68a6b786acf91a311c78444b1dc4f97c539f9f78de22a", - "zh:76858f9d8b460e7b2a338c477671d07286b0d287fd2d2e3214030ae8f61dd56e", - "zh:a13b69fb43cb8746793b3069c4d897bb18f454290b496f19d03c3387d1c9a2dc", - "zh:a90ca81bb9bb509063b736842250ecff0f886a91baae8de65c8430168001dad9", - "zh:c4de401395936e41234f1956ebadbd2ed9f414e6908f27d578614aaa529870d4", - "zh:c657e121af8fde19964482997f0de2d5173217274f6997e16389e7707ed8ece8", - "zh:d68b07a67fbd604c38ec9733069fbf23441436fecf554de6c75c032f82e1ef19", + "h1:j/BqLS2N2AScZyotd9nZpHdieJ7e5S8y+A+ZfIu8kL8=", + "zh:0ab58d6f8991d436c7d2dbd89ed814709b949b07ac5a54ee53b0aec1fa772a8b", + "zh:60b347abcb56f45d97c56f14d895069cd15a83993f199777f571b79fea3642ee", + "zh:6889be32640349230de3f23856e6f04e0e9ced4a84a27d3f552fa54684448218", + "zh:73f8e1ecf7135033165fb14b7e8bf4d656f3ce13065ec35762ea0481975328c7", + "zh:94ce25ee253eca0b42cae9c856b36bca8103b6453012d1b279c3623c805f2d42", + "zh:96bc6de9fd67bc446fd11257872e1ffb1029a996ed1d65a3f6b43f6d408ad9ab", + "zh:97c609a310a51bfd504d704e036d72064a84bf0bdb36cc08cd4cc66098212b41", + "zh:a12c16e94533c5bd123f75032576b9dc91dd5d5ccd5f7cf331d0f2e1adc55cf8", + "zh:c4f014f876adf7af57188795050bda5b0029d8c7d7773031102b6c36dcf1fc21", + "zh:d9b0a21583aaa3df3a95394fb949a3c515ff71c2ff5a1fc4a73d364aa90bfca5", + "zh:da510d22f0c6d71ad19a76406f106b782448f512375787ecfabb338ed1e311a7", + "zh:f0e9447a9ce3a24cdaa113089e65663c836d8b9bfdb915a1c0284e0112cab5c0", "zh:f569b65999264a9416862bca5cd2a6177d94ccb0424f3a4ef424428912b9cb3c", ] } diff --git a/terraform/eks/asg_tags.tf b/terraform/eks/asg_tags.tf deleted file mode 100644 index 3c05429..0000000 --- a/terraform/eks/asg_tags.tf +++ /dev/null @@ -1,67 +0,0 @@ -# Licensed to the Apache Software Foundation (ASF) under one -# or more contributor license agreements. See the NOTICE file -# distributed with this work for additional information -# regarding copyright ownership. The ASF licenses this file -# to you under the Apache License, Version 2.0 (the -# "License"); you may not use this file except in compliance -# with the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, -# software distributed under the License is distributed on an -# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -# KIND, either express or implied. See the License for the -# specific language governing permissions and limitations -# under the License. - -###################################################################################################### -# In order to scale the node groups from zero, we need to add the labels and taints to the ASGs tags: -# k8s.io/cluster-autoscaler/node-template/label/<label-key> = <label-value> -# k8s.io/cluster-autoscaler/node-template/taint/<taint-key> = <taint-value>:<taint-effect> -###################################################################################################### - -locals { - - taint_effects = { - NO_SCHEDULE = "NoSchedule" - NO_EXECUTE = "NoExecute" - PREFER_NO_SCHEDULE = "PreferNoSchedule" - } - - cluster_autoscaler_label_tags = merge([ - for name, group in module.eks.eks_managed_node_groups : { - for label_name, label_value in coalesce(group.node_group_labels, {}) : "${name}|label|${label_name}" => { - autoscaling_group = group.node_group_autoscaling_group_names[0], - key = "k8s.io/cluster-autoscaler/node-template/label/${label_name}", - value = label_value, - } - } - ]...) - - cluster_autoscaler_taint_tags = merge([ - for name, group in module.eks.eks_managed_node_groups : { - for taint in coalesce(group.node_group_taints, []) : "${name}|taint|${taint.key}" => { - autoscaling_group = group.node_group_autoscaling_group_names[0], - key = "k8s.io/cluster-autoscaler/node-template/taint/${taint.key}" - value = "${taint.value}:${local.taint_effects[taint.effect]}" - } - } - ]...) - - cluster_autoscaler_asg_tags = merge(local.cluster_autoscaler_label_tags, local.cluster_autoscaler_taint_tags) -} - -resource "aws_autoscaling_group_tag" "cluster_autoscaler_label_tags" { - for_each = local.cluster_autoscaler_asg_tags - - autoscaling_group_name = each.value.autoscaling_group - - tag { - key = each.value.key - value = each.value.value - - propagate_at_launch = true - } -} - diff --git a/terraform/eks/eks.tf b/terraform/eks/eks.tf index 5b4203a..bd901db 100644 --- a/terraform/eks/eks.tf +++ b/terraform/eks/eks.tf @@ -18,218 +18,59 @@ data "aws_availability_zones" "available" {} locals { - cluster_name = "airflow" + cluster_name = "airflow-ci" } module "eks" { source = "terraform-aws-modules/eks/aws" - version = "20.24.0" + version = "~> 21.24" - cluster_name = local.cluster_name - cluster_version = "1.30" + name = local.cluster_name + kubernetes_version = var.kubernetes_version vpc_id = module.vpc.vpc_id subnet_ids = module.vpc.public_subnets - cluster_endpoint_public_access = true - - eks_managed_node_group_defaults = { - ami_type = "AL2_ARM_64" - use_custom_launch_template = false - disk_size = 50 - desired_size = 0 - } - - eks_managed_node_groups = { - - default_nodes = { - name = "default" - - instance_types = [ - "t4g.medium", - "t4g.large", - "t4g.xlarge", - "m7g.medium", - "m7g.large", - "m7g.xlarge", - ] - - min_size = 0 - max_size = 5 - desired_size = 1 - - capacity_type = "SPOT" - - labels = { - "node-type" = "default" - } + endpoint_public_access = true + enable_cluster_creator_admin_permissions = true + + addons = { + coredns = {} + kube-proxy = {} + # before_compute: the module disables EKS's bootstrap self-managed CNI when + # managed addons are used, so vpc-cni must exist BEFORE nodes boot — nodes + # without a CNI never become Ready and the node group lands in CREATE_FAILED + # (bitten by this on 2026-07-25 when a crashed apply skipped the addons). + vpc-cni = { + before_compute = true } - - # GHA arm runners' nodes - GHA_runners_small = { - name = "gha-runners-small" - - instance_types = [for node_type in var.runners_node_types: "${node_type}.${var.small_runners_node_size}"] - - min_size = 0 - max_size = 30 - - capacity_type = "SPOT" - - disk_size = 50 - - labels = { - "node-type" = "gha-runners" - "size" = "small" - "arch" = "arm64" - } - - taints = [ - { - "key" = "node-type" - "value" = "gha-runners" - "effect" = "NO_SCHEDULE" - } - ] + # Required for Karpenter's Pod Identity association. + eks-pod-identity-agent = { + before_compute = true } + } - GHA_runners_medium = { - name = "gha-runners-medium" - - instance_types = [for node_type in var.runners_node_types: "${node_type}.${var.medium_runners_node_size}"] - - min_size = 0 - max_size = 30 - - capacity_type = "SPOT" - - disk_size = 50 - - labels = { - "node-type" = "gha-runners" - "size" = "medium" - "arch" = "arm64" - } - - taints = [ - { - "key" = "node-type" - "value" = "gha-runners" - "effect" = "NO_SCHEDULE" - } - ] - } - - GHA_runners_large = { - name = "gha-runners-large" - - instance_types = [for node_type in var.runners_node_types: "${node_type}.${var.large_runners_node_size}"] - - min_size = 0 - max_size = 30 - - capacity_type = "SPOT" - - labels = { - "node-type" = "gha-runners" - "size" = "large" - "arch" = "arm64" - } - - taints = [ - { - "key" = "node-type" - "value" = "gha-runners" - "effect" = "NO_SCHEDULE" - } - ] - } - - # GHA x64 runners' nodes - x64_GHA_runners_small = { - name = "gha-x64-runners-small" - - ami_type = "AL2_x86_64" - - instance_types = [for node_type in var.x64_runners_node_types: "${node_type}.${var.small_runners_x64_node_size}"] - - min_size = 0 - max_size = 30 - - capacity_type = "SPOT" - - disk_size = 50 - - labels = { - "node-type" = "gha-runners" - "size" = "small" - "arch" = "x64" - } - - taints = [ - { - "key" = "node-type" - "value" = "gha-runners" - "effect" = "NO_SCHEDULE" - } - ] - } - - x64_GHA_runners_medium = { - name = "gha-x64-runners-medium" - - ami_type = "AL2_x86_64" - - instance_types = [for node_type in var.x64_runners_node_types: "${node_type}.${var.medium_runners_node_size}"] - - min_size = 0 - max_size = 30 - - capacity_type = "SPOT" - - disk_size = 50 - - labels = { - "node-type" = "gha-runners" - "size" = "medium" - "arch" = "x64" - } - - taints = [ - { - "key" = "node-type" - "value" = "gha-runners" - "effect" = "NO_SCHEDULE" - } - ] - } - - x64_GHA_runners_large = { - name = "gha-x64-runners-large" - - ami_type = "AL2_x86_64" - - instance_types = [for node_type in var.x64_runners_node_types: "${node_type}.${var.large_runners_node_size}"] - - min_size = 0 - max_size = 30 + # Karpenter's EC2NodeClass discovers the node security group by this tag. + node_security_group_tags = { + "karpenter.sh/discovery" = local.cluster_name + } - capacity_type = "SPOT" + eks_managed_node_groups = { + # Small always-on group for the controllers (Karpenter, ARC, listeners). + # Everything else runs on Karpenter-provisioned nodes. + system = { + name = "airflow-ci-system" + ami_type = "AL2023_ARM_64_STANDARD" + instance_types = var.system_node_instance_types + capacity_type = "ON_DEMAND" + + min_size = 2 + max_size = 3 + desired_size = 2 labels = { - "node-type" = "gha-runners" - "size" = "large" - "arch" = "x64" + "airflow-ci/role" = "system" } - - taints = [ - { - "key" = "node-type" - "value" = "gha-runners" - "effect" = "NO_SCHEDULE" - } - ] } } } - diff --git a/terraform/eks/iam.tf b/terraform/eks/iam.tf deleted file mode 100644 index e2c3000..0000000 --- a/terraform/eks/iam.tf +++ /dev/null @@ -1,134 +0,0 @@ -# Licensed to the Apache Software Foundation (ASF) under one -# or more contributor license agreements. See the NOTICE file -# distributed with this work for additional information -# regarding copyright ownership. The ASF licenses this file -# to you under the Apache License, Version 2.0 (the -# "License"); you may not use this file except in compliance -# with the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, -# software distributed under the License is distributed on an -# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -# KIND, either express or implied. See the License for the -# specific language governing permissions and limitations -# under the License. - -data "aws_iam_policy_document" "autoscaler_policy_document" { - statement { - effect = "Allow" - actions = [ - "autoscaling:DescribeAutoScalingGroups", - "autoscaling:DescribeAutoScalingInstances", - "autoscaling:DescribeLaunchConfigurations", - "autoscaling:DescribeScalingActivities", - "autoscaling:DescribeTags", - "ec2:DescribeInstanceTypes", - "ec2:DescribeLaunchTemplateVersions" - ] - resources = ["*"] - } - - statement { - effect = "Allow" - actions = [ - "autoscaling:SetDesiredCapacity", - "autoscaling:TerminateInstanceInAutoScalingGroup", - "ec2:DescribeImages", - "ec2:GetInstanceTypesFromInstanceRequirements", - "eks:DescribeNodegroup" - ] - resources = ["*"] - } -} - -resource "aws_iam_policy" "autoscaler_policy" { - name = "eks-autoscaler-policy" - description = "EKS Autoscaler Policy" - policy = data.aws_iam_policy_document.autoscaler_policy_document.json -} - -data "aws_iam_policy_document" "autoscaler_role_assume_policy_document" { - statement { - actions = [ - "sts:AssumeRoleWithWebIdentity" - ] - - principals { - type = "Federated" - identifiers = [ - module.eks.oidc_provider_arn - ] - } - - condition { - test = "StringEquals" - variable = "${module.eks.oidc_provider}:sub" - values = [ - "system:serviceaccount:infra:cluster-autoscaler" - ] - } - - condition { - test = "StringEquals" - variable = "${module.eks.oidc_provider}:aud" - values = [ - "sts.amazonaws.com" - ] - } - - } -} - -resource "aws_iam_role" "autoscaler_role" { - name = "eks-autoscaler-role" - assume_role_policy = data.aws_iam_policy_document.autoscaler_role_assume_policy_document.json -} - -resource "aws_iam_role_policy_attachment" "autoscaler_policy_attachment" { - role = aws_iam_role.autoscaler_role.name - policy_arn = aws_iam_policy.autoscaler_policy.arn -} - -data "aws_iam_policy_document" "ebs_csi_driver_assume_policy_document" { - statement { - actions = [ - "sts:AssumeRoleWithWebIdentity" - ] - - principals { - type = "Federated" - identifiers = [ - module.eks.oidc_provider_arn - ] - } - - condition { - test = "StringEquals" - variable = "${module.eks.oidc_provider}:sub" - values = [ - "system:serviceaccount:kube-system:ebs-csi-controller-sa" - ] - } - - condition { - test = "StringEquals" - variable = "${module.eks.oidc_provider}:aud" - values = [ - "sts.amazonaws.com" - ] - } - - } -} - -resource "aws_iam_role" "ebs_csi_driver" { - name = "ebs-csi-driver" - assume_role_policy = data.aws_iam_policy_document.ebs_csi_driver_assume_policy_document.json -} - -resource "aws_iam_role_policy_attachment" "ebs_csi_driver" { - role = aws_iam_role.ebs_csi_driver.name - policy_arn = "arn:aws:iam::aws:policy/service-role/AmazonEBSCSIDriverPolicy" -} diff --git a/terraform/eks/vpc.tf b/terraform/eks/karpenter.tf similarity index 50% copy from terraform/eks/vpc.tf copy to terraform/eks/karpenter.tf index 1d80025..c334d69 100644 --- a/terraform/eks/vpc.tf +++ b/terraform/eks/karpenter.tf @@ -15,30 +15,22 @@ # specific language governing permissions and limitations # under the License. -module "vpc" { - source = "terraform-aws-modules/vpc/aws" - version = "5.13.0" +# IAM + SQS plumbing for Karpenter (controller role via EKS Pod Identity, node +# role + access entry, spot-interruption queue). Karpenter itself is installed +# with helmfile (helm/infra-helmfile.yaml); the names below are referenced from +# the helm values, so they are pinned instead of name-prefixed. +module "karpenter" { + source = "terraform-aws-modules/eks/aws//modules/karpenter" + version = "~> 21.24" - name = "airflow-vpc" + cluster_name = module.eks.cluster_name - cidr = "10.1.0.0/16" - azs = slice(data.aws_availability_zones.available.names, 0, 3) + # Referenced by helm/values/karpenter.yaml (settings.interruptionQueue). + queue_name = "airflow-ci-karpenter" - private_subnets = ["10.1.1.0/24", "10.1.2.0/24", "10.1.3.0/24"] - public_subnets = ["10.1.4.0/24", "10.1.5.0/24", "10.1.6.0/24"] + # Referenced by the EC2NodeClass (spec.role) in helm/karpenter-resources. + node_iam_role_name = "airflow-ci-runner-node" + node_iam_role_use_name_prefix = false - enable_nat_gateway = false - enable_dns_hostnames = true - - map_public_ip_on_launch = true - - public_subnet_tags = { - "kubernetes.io/cluster/${local.cluster_name}" = "shared" - "kubernetes.io/role/elb" = 1 - } - - private_subnet_tags = { - "kubernetes.io/cluster/${local.cluster_name}" = "shared" - "kubernetes.io/role/internal-elb" = 1 - } -} \ No newline at end of file + create_pod_identity_association = true +} diff --git a/terraform/eks/vpc.tf b/terraform/eks/outputs.tf similarity index 50% copy from terraform/eks/vpc.tf copy to terraform/eks/outputs.tf index 1d80025..f7d224d 100644 --- a/terraform/eks/vpc.tf +++ b/terraform/eks/outputs.tf @@ -15,30 +15,27 @@ # specific language governing permissions and limitations # under the License. -module "vpc" { - source = "terraform-aws-modules/vpc/aws" - version = "5.13.0" - - name = "airflow-vpc" - - cidr = "10.1.0.0/16" - azs = slice(data.aws_availability_zones.available.names, 0, 3) - - private_subnets = ["10.1.1.0/24", "10.1.2.0/24", "10.1.3.0/24"] - public_subnets = ["10.1.4.0/24", "10.1.5.0/24", "10.1.6.0/24"] - - enable_nat_gateway = false - enable_dns_hostnames = true - - map_public_ip_on_launch = true - - public_subnet_tags = { - "kubernetes.io/cluster/${local.cluster_name}" = "shared" - "kubernetes.io/role/elb" = 1 - } - - private_subnet_tags = { - "kubernetes.io/cluster/${local.cluster_name}" = "shared" - "kubernetes.io/role/internal-elb" = 1 - } -} \ No newline at end of file +output "cluster_name" { + description = "EKS cluster name" + value = module.eks.cluster_name +} + +output "cluster_endpoint" { + description = "EKS API endpoint" + value = module.eks.cluster_endpoint +} + +output "karpenter_node_iam_role_name" { + description = "IAM role assumed by Karpenter-launched nodes (EC2NodeClass spec.role)" + value = module.karpenter.node_iam_role_name +} + +output "karpenter_interruption_queue" { + description = "SQS queue for spot interruption handling (karpenter settings.interruptionQueue)" + value = module.karpenter.queue_name +} + +output "update_kubeconfig" { + description = "Command to configure kubectl" + value = "aws eks --region ${var.region} update-kubeconfig --name ${module.eks.cluster_name} --alias ${module.eks.cluster_name}" +} diff --git a/terraform/eks/provider.tf b/terraform/eks/provider.tf index 89656fe..5388c88 100644 --- a/terraform/eks/provider.tf +++ b/terraform/eks/provider.tf @@ -15,7 +15,16 @@ # specific language governing permissions and limitations # under the License. +# Credentials come from the environment (AWS_PROFILE locally, OIDC role in a +# future GitHub Actions workflow) — do not hardcode a profile here. provider "aws" { - profile = "airflow" - region = var.region + region = var.region + + default_tags { + tags = { + Project = "airflow-ci" + ManagedBy = "terraform" + Repo = "apache/airflow-ci-infra" + } + } } \ No newline at end of file diff --git a/terraform/eks/runner_pod_identity.tf b/terraform/eks/runner_pod_identity.tf new file mode 100644 index 0000000..bbd267d --- /dev/null +++ b/terraform/eks/runner_pod_identity.tf @@ -0,0 +1,113 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +# The airflow CI workflows pass artifacts between jobs via the S3-backed stash +# actions (.github/actions/stash-s3-*). Runner pods get scoped S3 access +# through EKS Pod Identity — node IAM credentials are deliberately +# unreachable from pods (IMDS hop limit 1 on the EC2NodeClass). +locals { + # Same-region successor of the us-east-1 `airflow-ci-runner-cache` bucket + # (bucket names are global, so the regional suffix disambiguates). Contents + # are disposable inter-job artifacts — nothing was migrated. + stash_bucket = "airflow-ci-runner-cache-us-east-2" + + # One service account per runner scale set, created by the gha-runner-scale-set + # chart as "<scale-set-name>-gha-rs-no-permission" in the arc-runners namespace. + runner_scale_set_names = [ + "airflow-runner-amd-small", + "airflow-runner-amd-medium", + "airflow-runner-amd-xl", + "airflow-runner-arm-small", + "airflow-runner-arm-medium", + "airflow-runner-arm-xl", + ] +} + +resource "aws_s3_bucket" "stash" { + bucket = local.stash_bucket +} + +resource "aws_s3_bucket_public_access_block" "stash" { + bucket = aws_s3_bucket.stash.id + + block_public_acls = true + block_public_policy = true + ignore_public_acls = true + restrict_public_buckets = true +} + +# Stashes are short-lived hand-offs between jobs of a run; the actions ask for +# ~2-day retention, the lifecycle rule is the backstop that actually deletes. +resource "aws_s3_bucket_lifecycle_configuration" "stash" { + bucket = aws_s3_bucket.stash.id + + rule { + id = "expire-stashes" + status = "Enabled" + + filter {} + + expiration { + days = 3 + } + + abort_incomplete_multipart_upload { + days_after_initiation = 1 + } + } +} + +data "aws_iam_policy_document" "runner_pods_assume" { + statement { + actions = ["sts:AssumeRole", "sts:TagSession"] + principals { + type = "Service" + identifiers = ["pods.eks.amazonaws.com"] + } + } +} + +data "aws_iam_policy_document" "runner_pods_stash" { + statement { + actions = ["s3:GetObject", "s3:PutObject"] + resources = ["arn:aws:s3:::${local.stash_bucket}/*"] + } + statement { + actions = ["s3:ListBucket", "s3:GetBucketLocation"] + resources = ["arn:aws:s3:::${local.stash_bucket}"] + } +} + +resource "aws_iam_role" "runner_pods" { + name = "airflow-ci-runner-pods" + assume_role_policy = data.aws_iam_policy_document.runner_pods_assume.json +} + +resource "aws_iam_role_policy" "runner_pods_stash" { + name = "stash-bucket-access" + role = aws_iam_role.runner_pods.id + policy = data.aws_iam_policy_document.runner_pods_stash.json +} + +resource "aws_eks_pod_identity_association" "runner_pods" { + for_each = toset(local.runner_scale_set_names) + + cluster_name = module.eks.cluster_name + namespace = "arc-runners" + service_account = "${each.value}-gha-rs-no-permission" + role_arn = aws_iam_role.runner_pods.arn +} diff --git a/terraform/eks/state.tf b/terraform/eks/state.tf index 025e912..8801ae2 100644 --- a/terraform/eks/state.tf +++ b/terraform/eks/state.tf @@ -18,10 +18,16 @@ terraform { backend "s3" { - region = "us-east-2" - bucket = "airflow-ci-tfstate" - key = "tf-state/eks.tfstate" - encrypt = true - dynamodb_table = "TerraformLocks" + region = "us-east-2" + bucket = "airflow-ci-tfstate" + # Fresh key for the 2026 airflow-ci cluster. tf-state/eks.tfstate is the + # 2024 deployment's state and is deliberately left untouched (its cluster + # is gone but the VPC/IAM leftovers still exist and must not be disturbed). + key = "tf-state/eks-airflow-ci.tfstate" + encrypt = true + # S3-native locking (.tflock object next to the state). Replaces the + # deprecated dynamodb_table locking; the TerraformLocks DynamoDB table is + # still referenced by the 2024 state and must not be deleted. + use_lockfile = true } } diff --git a/terraform/eks/variables.tf b/terraform/eks/variables.tf index 045c496..deeb5a0 100644 --- a/terraform/eks/variables.tf +++ b/terraform/eks/variables.tf @@ -21,48 +21,14 @@ variable "region" { default = "us-east-2" } -variable "runners_node_types" { - description = "Node type for the runners" - type = list(string) - default = [ - "t4g", - "m7g", - "m6g", - ] -} - -variable "x64_runners_node_types" { - description = "X64 node type for the runners" - type = list(string) - default = [ - "m7a", - "m6a", - "t3", - "t3a", - ] -} - -variable "small_runners_node_size" { - description = "Node size for the small runners" - type = string - default = "medium" -} - -variable "small_runners_x64_node_size" { - # This is a separate variable because the r series doesn't have a medium size - description = "Node size for the small runners" +variable "kubernetes_version" { + description = "EKS Kubernetes version; keep within standard support to avoid extended-support fees" type = string - default = "large" + default = "1.34" } -variable "medium_runners_node_size" { - description = "Node size for the medium runners" - type = string - default = "xlarge" +variable "system_node_instance_types" { + description = "Instance types for the small always-on system node group (Karpenter + ARC controllers)" + type = list(string) + default = ["t4g.medium"] } - -variable "large_runners_node_size" { - description = "Node size for the large runners" - type = string - default = "2xlarge" -} \ No newline at end of file diff --git a/terraform/eks/version.tf b/terraform/eks/version.tf index a215415..93977fd 100644 --- a/terraform/eks/version.tf +++ b/terraform/eks/version.tf @@ -19,8 +19,9 @@ terraform { required_providers { aws = { source = "hashicorp/aws" - version = "~> 5.64.0" + version = "~> 6.0" } } - required_version = "~> 1.9.5" + # 1.10+ for S3-native state locking (use_lockfile) + required_version = ">= 1.10.0" } \ No newline at end of file diff --git a/terraform/eks/vpc.tf b/terraform/eks/vpc.tf index 1d80025..3ff2339 100644 --- a/terraform/eks/vpc.tf +++ b/terraform/eks/vpc.tf @@ -15,17 +15,21 @@ # specific language governing permissions and limitations # under the License. +# Nodes live in public subnets with public IPs on purpose: a NAT gateway costs +# ~$400/yr while public IPv4 on ephemeral runner nodes is a few $/mo, and the +# security groups allow no inbound traffic (runners are outbound-only). module "vpc" { source = "terraform-aws-modules/vpc/aws" - version = "5.13.0" + version = "~> 6.6" - name = "airflow-vpc" + name = "airflow-ci-vpc" cidr = "10.1.0.0/16" azs = slice(data.aws_availability_zones.available.names, 0, 3) - private_subnets = ["10.1.1.0/24", "10.1.2.0/24", "10.1.3.0/24"] - public_subnets = ["10.1.4.0/24", "10.1.5.0/24", "10.1.6.0/24"] + # Docker inside the runners carves networks out of 172.16.0.0/14 — the VPC + # range must stay clear of it, hence 10.x. + public_subnets = ["10.1.4.0/24", "10.1.5.0/24", "10.1.6.0/24"] enable_nat_gateway = false enable_dns_hostnames = true @@ -34,11 +38,7 @@ module "vpc" { public_subnet_tags = { "kubernetes.io/cluster/${local.cluster_name}" = "shared" - "kubernetes.io/role/elb" = 1 + # Karpenter's EC2NodeClass discovers subnets by this tag. + "karpenter.sh/discovery" = local.cluster_name } - - private_subnet_tags = { - "kubernetes.io/cluster/${local.cluster_name}" = "shared" - "kubernetes.io/role/internal-elb" = 1 - } -} \ No newline at end of file +}
