This is an automated email from the ASF dual-hosted git repository. potiuk pushed a commit to branch runs-on-on-demand-share-report in repository https://gitbox.apache.org/repos/asf/airflow-ci-infra.git
commit 9daed77eb0eb0fd33c853b2ac757dbac7d2c18ae Author: Jarek Potiuk <[email protected]> AuthorDate: Mon Oct 26 21:07:22 2026 +0100 Warn weekly when RunsOn runs mostly on on-demand instances Generated-by: Claude Opus 5 --- .github/workflows/runs-on-usage-report.yml | 103 +++++++++++++++++++++++++++++ terraform/runs-on-access/README.rst | 6 +- terraform/runs-on-access/main.tf | 25 ++++++- terraform/runs-on/README.rst | 5 ++ 4 files changed, 137 insertions(+), 2 deletions(-) diff --git a/.github/workflows/runs-on-usage-report.yml b/.github/workflows/runs-on-usage-report.yml new file mode 100644 index 0000000..d949e14 --- /dev/null +++ b/.github/workflows/runs-on-usage-report.yml @@ -0,0 +1,103 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +--- +# Weekly check of how much of the account's EC2 vCPU usage in us-east-2 ran on on-demand instead of +# Spot. Above the threshold it keeps one issue open and comments the week's numbers on it; once the +# share is back under the threshold it closes the issue. The AWS role is runs-on-usage-report from +# terraform/runs-on-access. +name: RunsOn on-demand share + +on: # yamllint disable-line rule:truthy + schedule: + - cron: "0 6 * * 1" + workflow_dispatch: + +permissions: {} + +concurrency: + group: runs-on-usage-report + +jobs: + report: + if: github.repository == 'apache/airflow-ci-infra' + runs-on: ubuntu-24.04 + permissions: + id-token: write + issues: write + steps: + - uses: aws-actions/configure-aws-credentials@e1253824e5c10ff9df46874f81ed3ec929e19cfd # v6.3.0 + with: + role-to-assume: arn:aws:iam::827901512104:role/runs-on-usage-report + aws-region: us-east-2 + - name: Report the on-demand share of the last 7 days + env: + GH_TOKEN: ${{ github.token }} + GH_REPO: ${{ github.repository }} + THRESHOLD_PERCENT: "30" + ISSUE_TITLE: "RunsOn runners ran on on-demand instances for more than 30% of their vCPU-hours" + run: | + end=$(date -u +%Y-%m-%dT%H:00:00Z) + start=$(date -u -d "${end} - 7 days" +%Y-%m-%dT%H:00:00Z) + # The hourly average of the vCPUs in use is the vCPU-hours of that hour; hours without + # any instance of a class have no datapoint. + query() { + jq -nc --arg id "$1" --arg class "$2" '{Id: $id, MetricStat: {Metric: { + Namespace: "AWS/Usage", MetricName: "ResourceCount", Dimensions: [ + {Name: "Service", Value: "EC2"}, {Name: "Type", Value: "Resource"}, + {Name: "Resource", Value: "vCPU"}, {Name: "Class", Value: $class}]}, + Period: 3600, Stat: "Average"}}' + } + queries="[$(query ondemand Standard/OnDemand),$(query spot Standard/Spot)]" + # A command substitution, so that a failed call fails the job instead of reading as zero. + data=$(aws cloudwatch get-metric-data --start-time "${start}" --end-time "${end}" \ + --metric-data-queries "${queries}" --output json) + incomplete=$(jq -r '[.MetricDataResults[] | select(.StatusCode != "Complete") | .Id] + | join(" ")' <<< "${data}") + if [[ -n "${incomplete}" ]] || [[ $(jq '.MetricDataResults | length' <<< "${data}") != 2 ]]; then + echo "::error::Incomplete CloudWatch results (${incomplete:-missing queries})" + exit 1 + fi + read -r on_demand spot <<< "$(jq -r '.MetricDataResults + | map({(.Id): (.Values | add // 0 | round)}) | add | "\(.ondemand) \(.spot)"' <<< "${data}")" + if (( on_demand + spot == 0 )); then + echo "::error::No EC2 vCPU usage found between ${start} and ${end}" + exit 1 + fi + share=$(( 100 * on_demand / (on_demand + spot) )) + summary="From ${start} to ${end}: ${on_demand} on-demand and ${spot} Spot vCPU-hours," + summary+=" ${share}% on-demand (threshold ${THRESHOLD_PERCENT}%)." + echo "${summary}" | tee -a "${GITHUB_STEP_SUMMARY}" + + issue=$(gh issue list --state open --search "\"${ISSUE_TITLE}\" in:title" \ + --json number,title --jq ".[] | select(.title == \"${ISSUE_TITLE}\") | .number" | head -1) + # Compared without the rounding of the displayed share, which 30.5% would hide. + if (( 100 * on_demand > THRESHOLD_PERCENT * (on_demand + spot) )); then + if [[ -n "${issue}" ]]; then + gh issue comment "${issue}" --body "${summary}" + else + gh issue create --title "${ISSUE_TITLE}" --body "${summary} + + The RunsOn stack launches Spot first and falls back to on-demand when Spot capacity is + short or while its Spot circuit breaker is open (\`SpotCircuitBreaker\` in + \`terraform/runs-on/main.tf\`). On-demand instances cost more and count against the + smaller on-demand vCPU quota, so jobs queue once it is full. The RunsOn CloudWatch + dashboard shows the circuit breaker state. This issue gets a comment every week while the + share stays above ${THRESHOLD_PERCENT}%, and is closed once it is back under." + fi + elif [[ -n "${issue}" ]]; then + gh issue close "${issue}" --comment "${summary}" + fi diff --git a/terraform/runs-on-access/README.rst b/terraform/runs-on-access/README.rst index 54377cb..04979cd 100644 --- a/terraform/runs-on-access/README.rst +++ b/terraform/runs-on-access/README.rst @@ -54,6 +54,10 @@ Roles needs that list extended here first. A template release that adds an IAM resource with a new logical-ID prefix fails its update with ``AccessDenied`` until the list in ``main.tf`` is extended. +``runs-on-usage-report`` + Assumed through GitHub OIDC by workflows running on ``main`` of ``apache/airflow-ci-infra``, + for the weekly ``.github/workflows/runs-on-usage-report.yml``. It can only call + ``cloudwatch:GetMetricData``, to compare the account's on-demand and Spot vCPU usage. All trust policies pin the repository by its immutable GitHub IDs (``repository_owner_id`` ``47359`` and ``repository_id`` ``321691041``), so a renamed or re-created repository cannot match @@ -96,7 +100,7 @@ Bootstrap order cd terraform/runs-on-access terraform init - terraform plan -out=runs-on-access.tfplan # 3 roles and their policies, nothing else + terraform plan -out=runs-on-access.tfplan # 4 roles and their policies, nothing else terraform apply runs-on-access.tfplan 5. The same admin applies ``terraform/runs-on`` once by hand, which sets ``iam_role_arn`` on the diff --git a/terraform/runs-on-access/main.tf b/terraform/runs-on-access/main.tf index 97fece8..f1df0f0 100644 --- a/terraform/runs-on-access/main.tf +++ b/terraform/runs-on-access/main.tf @@ -43,6 +43,8 @@ locals { ]) apply = [for s in local.github_subject_prefixes : "${s}:environment:runs-on-prod"] } + # The weekly report runs on a schedule, so only main. + usage_report_subjects = [for s in local.github_subject_prefixes : "${s}:ref:refs/heads/main"] # IAM names the RunsOn template generates for the runs-on stack: <stack>-<logical id>-<suffix>. # Matching the logical-ID prefixes rather than "runs-on-*" keeps every other role in the account, @@ -68,7 +70,7 @@ data "aws_iam_openid_connect_provider" "github" { } data "aws_iam_policy_document" "github_trust" { - for_each = local.github_subjects + for_each = merge(local.github_subjects, { usage-report = local.usage_report_subjects }) statement { actions = ["sts:AssumeRoleWithWebIdentity"] @@ -199,6 +201,27 @@ resource "aws_iam_role_policy" "apply" { policy = data.aws_iam_policy_document.apply.json } +# Read-only role for .github/workflows/runs-on-usage-report.yml, which compares the account's +# on-demand and Spot vCPU usage. GetMetricData has no resource-level permissions. +resource "aws_iam_role" "usage_report" { + name = "runs-on-usage-report" + description = "Weekly on-demand share report of apache/airflow-ci-infra GitHub Actions" + assume_role_policy = data.aws_iam_policy_document.github_trust["usage-report"].json +} + +data "aws_iam_policy_document" "usage_report" { + statement { + actions = ["cloudwatch:GetMetricData"] + resources = ["*"] + } +} + +resource "aws_iam_role_policy" "usage_report" { + name = "usage-report" + role = aws_iam_role.usage_report.name + policy = data.aws_iam_policy_document.usage_report.json +} + data "aws_iam_policy_document" "cloudformation_trust" { statement { actions = ["sts:AssumeRole"] diff --git a/terraform/runs-on/README.rst b/terraform/runs-on/README.rst index 061c82e..1ff7026 100644 --- a/terraform/runs-on/README.rst +++ b/terraform/runs-on/README.rst @@ -187,6 +187,11 @@ Observability - CloudWatch dashboard and log groups above; ``roc`` CLI (`docs <https://runs-on.com/docs/observability/cli/>`_): ``roc stack doctor``, ``roc stack logs``, ``roc logs <job url> --watch``, ``roc connect <job url>`` over SSM (``SSMAllowed=true``). +- ``.github/workflows/runs-on-usage-report.yml`` checks every Monday how much of the last week's + EC2 vCPU-hours in us-east-2 ran on on-demand instances. Above 30% it keeps one issue open in + this repository and comments each week's numbers on it, and it closes the issue once the share + is back under. The numbers are account-wide (``AWS/Usage`` metrics), so any other EC2 + instances in the region count too. - Optional OTLP export (``OtelExporterEndpoint`` / ``OtelExporterHeaders`` + ``extras=otel`` on the label) to SigNoz or any OTLP backend; RunsOn publishes a SigNoz dashboard. Not deployed by the stack.
