This is an automated email from the ASF dual-hosted git repository.
kaxil pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/airflow.git
The following commit(s) were added to refs/heads/main by this push:
new 16f974f6a35 Run the `common.ai` agent framework adapter tests in CI
(#73912)
16f974f6a35 is described below
commit 16f974f6a35388c867bd1da1c2a3bf3d93e9c6cc
Author: Kaxil Naik <[email protected]>
AuthorDate: Wed Sep 30 18:52:06 2026 +0100
Run the `common.ai` agent framework adapter tests in CI (#73912)
* Run the common.ai agent framework adapter tests in CI
Strands Agents caps mcp below the version uv.lock resolves, so it cannot
be installed in the workspace and the common.ai adapter tests for it were
skipped in every CI job.
A new job installs the framework into the CI image and runs the tests of
the framework-neutral tools and their adapters. Outside a canary run every
package already in the image is held at its version with uv's --override,
so the adapters are tested against the same dependencies as the rest of
Airflow. On a canary run the framework's own pins win, which is what a user
installing it gets. The job runs on main when common.ai or common.sql code,
the common.ai dependencies or uv.lock change. It is not a dependency of
finalize-tests, so a breaking framework release does not stop the image
cache push, and notify-slack waits for it so a canary failure is reported.
* Add Google ADK to the agent framework CI job
ADK caps opentelemetry and websockets below the locked versions, the same
way
Strands caps mcp, so its adapter tests were skipped everywhere too. Both
modes
of the job install it now and check that it imports.
* Run the agent framework tests one framework per job, on amd64 only
Each matrix entry installs a single framework, so a breaking release of one
cannot hide the other's result, and the entries run in parallel. The job is
gated to amd64 inside the reusable workflow, since ci-amd.yml and ci-arm.yml
must stay in sync.
* Tighten the agent framework test script after review
Drop the one-element framework array, fail with a message when the installed
framework is missing from the freeze output instead of dying silently under
pipefail, and document run-agent-framework-tests in the selective checks
table.
---
.github/workflows/agent-framework-tests.yml | 89 +++++++++++++++++++++
.github/workflows/ci-amd.yml | 19 +++++
.github/workflows/ci-arm.yml | 19 +++++
dev/breeze/doc/ci/04_selective_checks.md | 1 +
dev/breeze/doc/ci/05_workflows.md | 26 +++++++
.../src/airflow_breeze/utils/selective_checks.py | 18 +++++
dev/breeze/tests/test_selective_checks.py | 40 ++++++++++
scripts/in_container/run_agent_framework_tests.sh | 90 ++++++++++++++++++++++
8 files changed, 302 insertions(+)
diff --git a/.github/workflows/agent-framework-tests.yml
b/.github/workflows/agent-framework-tests.yml
new file mode 100644
index 00000000000..6e9de1174e9
--- /dev/null
+++ b/.github/workflows/agent-framework-tests.yml
@@ -0,0 +1,89 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+---
+name: Agent framework tests
+on: # yamllint disable-line rule:truthy
+ workflow_call:
+ inputs:
+ runners:
+ description: "The array of labels (in json form) determining runners."
+ required: true
+ type: string
+ platform:
+ description: "Platform for the build - 'linux/amd64' or 'linux/arm64'"
+ required: true
+ type: string
+ default-python-version:
+ description: "Which version of python should be used by default"
+ required: true
+ type: string
+ use-uv:
+ description: "Whether to use uv"
+ required: true
+ type: string
+ canary-run:
+ description: >
+ On a canary run the frameworks' own dependency pins win, which is
the environment a
+ user who installs them gets. Otherwise every package in the CI image
keeps its version
+ and the frameworks' caps on them are overridden.
+ required: true
+ type: string
+permissions:
+ contents: read
+jobs:
+ tests:
+ timeout-minutes: 30
+ name: >-
+ Agent framework tests: ${{ matrix.framework }}
+ (${{ inputs.canary-run == 'true' && 'framework pins' || 'image versions'
}})
+ runs-on: ${{ fromJSON(inputs.runners) }}
+ # amd64 only for now: ci-amd.yml and ci-arm.yml must stay in sync, so the
platform gate lives here.
+ if: inputs.platform == 'linux/amd64'
+ strategy:
+ # One job per framework, so a breaking release of one does not hide the
other's result.
+ fail-fast: false
+ matrix:
+ framework: [strands-agents, google-adk]
+ env:
+ GITHUB_REPOSITORY: ${{ github.repository }}
+ GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+ GITHUB_USERNAME: ${{ github.actor }}
+ PLATFORM: "${{ inputs.platform }}"
+ PYTHON_MAJOR_MINOR_VERSION: "${{ inputs.default-python-version }}"
+ VERBOSE: "true"
+ FRAMEWORK_PINS_FLAG: "${{ inputs.canary-run == 'true' &&
'--framework-pins' || '' }}"
+ steps:
+ - name: "Cleanup repo"
+ shell: bash
+ run: sudo rm -rf ${GITHUB_WORKSPACE}/*
+ - name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
+ uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 #
v7.0.1
+ with:
+ persist-credentials: false
+ - name: "Prepare breeze & CI image: ${{ inputs.default-python-version }}"
+ uses: ./.github/actions/prepare_breeze_and_image
+ with:
+ platform: ${{ inputs.platform }}
+ python: ${{ inputs.default-python-version }}
+ use-uv: ${{ inputs.use-uv }}
+ make-mnt-writeable-and-cleanup: true
+ - name: "Run the common.ai agent framework adapter tests"
+ run: >
+ breeze shell --backend none --skip-db-tests
+ "bash /opt/airflow/scripts/in_container/run_agent_framework_tests.sh
+ ${{ matrix.framework }} ${FRAMEWORK_PINS_FLAG}"
diff --git a/.github/workflows/ci-amd.yml b/.github/workflows/ci-amd.yml
index 7f3f761ac5e..8e670fbebd2 100644
--- a/.github/workflows/ci-amd.yml
+++ b/.github/workflows/ci-amd.yml
@@ -130,6 +130,7 @@ jobs:
pull-request-labels: ${{ steps.source-run-info.outputs.pr-labels }}
python-versions-list-as-string: ${{
steps.selective-checks.outputs.python-versions-list-as-string }}
python-versions: ${{ steps.selective-checks.outputs.python-versions }}
+ run-agent-framework-tests: ${{
steps.selective-checks.outputs.run-agent-framework-tests }}
run-airflow-ctl-tests: ${{
steps.selective-checks.outputs.run-airflow-ctl-tests }}
run-airflow-ctl-integration-tests: ${{
steps.selective-checks.outputs.run-airflow-ctl-integration-tests }}
run-amazon-tests: ${{ steps.selective-checks.outputs.run-amazon-tests }}
@@ -1138,6 +1139,23 @@ jobs:
test-timeout: 20
if: needs.build-info.outputs.run-airflow-ctl-tests == 'true'
+ tests-agent-frameworks:
+ name: "Agent framework tests"
+ uses: ./.github/workflows/agent-framework-tests.yml
+ needs: [build-info, build-ci-images]
+ permissions:
+ contents: read
+ packages: read
+ with:
+ runners: ${{ needs.build-info.outputs.runner-type }}
+ platform: ${{ needs.build-info.outputs.platform }}
+ default-python-version: "${{
needs.build-info.outputs.default-python-version }}"
+ use-uv: ${{ needs.build-info.outputs.use-uv }}
+ canary-run: ${{ needs.build-info.outputs.canary-run }}
+ # Not a dependency of finalize-tests: on a canary run the frameworks' own
pins win, and a
+ # breaking framework release should not stop finalize-tests from pushing
the image cache.
+ if: needs.build-info.outputs.run-agent-framework-tests == 'true'
+
finalize-tests:
name: Finalize tests
permissions:
@@ -1197,6 +1215,7 @@ jobs:
needs:
- build-info
- finalize-tests
+ - tests-agent-frameworks
if: >-
always() &&
!cancelled() &&
diff --git a/.github/workflows/ci-arm.yml b/.github/workflows/ci-arm.yml
index 2f257f32c5a..88d7109c944 100644
--- a/.github/workflows/ci-arm.yml
+++ b/.github/workflows/ci-arm.yml
@@ -119,6 +119,7 @@ jobs:
pull-request-labels: ${{ steps.source-run-info.outputs.pr-labels }}
python-versions-list-as-string: ${{
steps.selective-checks.outputs.python-versions-list-as-string }}
python-versions: ${{ steps.selective-checks.outputs.python-versions }}
+ run-agent-framework-tests: ${{
steps.selective-checks.outputs.run-agent-framework-tests }}
run-airflow-ctl-tests: ${{
steps.selective-checks.outputs.run-airflow-ctl-tests }}
run-airflow-ctl-integration-tests: ${{
steps.selective-checks.outputs.run-airflow-ctl-integration-tests }}
run-amazon-tests: ${{ steps.selective-checks.outputs.run-amazon-tests }}
@@ -1127,6 +1128,23 @@ jobs:
test-timeout: 20
if: needs.build-info.outputs.run-airflow-ctl-tests == 'true'
+ tests-agent-frameworks:
+ name: "Agent framework tests"
+ uses: ./.github/workflows/agent-framework-tests.yml
+ needs: [build-info, build-ci-images]
+ permissions:
+ contents: read
+ packages: read
+ with:
+ runners: ${{ needs.build-info.outputs.runner-type }}
+ platform: ${{ needs.build-info.outputs.platform }}
+ default-python-version: "${{
needs.build-info.outputs.default-python-version }}"
+ use-uv: ${{ needs.build-info.outputs.use-uv }}
+ canary-run: ${{ needs.build-info.outputs.canary-run }}
+ # Not a dependency of finalize-tests: on a canary run the frameworks' own
pins win, and a
+ # breaking framework release should not stop finalize-tests from pushing
the image cache.
+ if: needs.build-info.outputs.run-agent-framework-tests == 'true'
+
finalize-tests:
name: Finalize tests
permissions:
@@ -1186,6 +1204,7 @@ jobs:
needs:
- build-info
- finalize-tests
+ - tests-agent-frameworks
if: >-
always() &&
!cancelled() &&
diff --git a/dev/breeze/doc/ci/04_selective_checks.md
b/dev/breeze/doc/ci/04_selective_checks.md
index 489d60a8f73..d413de863a6 100644
--- a/dev/breeze/doc/ci/04_selective_checks.md
+++ b/dev/breeze/doc/ci/04_selective_checks.md
@@ -599,6 +599,7 @@ GitHub Actions to pass the list of parameters to a command
to execute
| pyproject-toml-changed | When
pyproject.toml changed in the PR.
| false | |
| python-versions | List of python
versions to use for that build
| \['3.10'\] | |
| python-versions-list-as-string | Which versions of
MySQL to use for tests as space-separated string
| 3.10 | * |
+| run-agent-framework-tests | Whether the
common.ai agent framework adapter tests should be run ("true"/"false")
| true | |
| run-amazon-tests | Whether Amazon
tests should be run ("true"/"false")
| true | |
| run-api-codegen | Whether
"api-codegen" are needed to run ("true"/"false")
| true | |
| run-api-tests | Whether
"api-tests" are needed to run ("true"/"false")
| true | |
diff --git a/dev/breeze/doc/ci/05_workflows.md
b/dev/breeze/doc/ci/05_workflows.md
index 5d1f51c87eb..e62dc41a283 100644
--- a/dev/breeze/doc/ci/05_workflows.md
+++ b/dev/breeze/doc/ci/05_workflows.md
@@ -262,6 +262,7 @@ Here's what each workflow group does and when it runs:
| **Additional PROD Image Tests** | Final validation of production images
(AMD only) | Yes | Yes | Yes |
| **Kubernetes Tests** | Tests deployment in Kubernetes
environments | Yes | Yes | Yes (1) |
| **Distribution Tests** | Tests Task SDK and CLI tools (AMD only)
| Yes | Yes | Yes |
+| **Agent Framework Tests** | Tests Common AI's adapters for other
agent frameworks (4) | Yes | Yes | No |
| **Finalize Tests** | Publishes results and updates shared
resources | Yes | Yes (2) | Yes (2) |
#### AMD-Only Workflows
@@ -312,6 +313,31 @@ Special tests (integration and system tests) run
selectively:
- In canary runs for scheduled quality checks
- When dependency upgrades require thorough testing
+**`(4)` Agent Framework Tests**
+
+The [Common AI provider](../../../../providers/common/ai/docs/index.rst) lets
an agent built with
+another framework use Airflow's toolsets, through small adapters in
+[`airflow.providers.common.ai.tools`](../../../../providers/common/ai/src/airflow/providers/common/ai/tools).
+The adapter tests import the framework they adapt, so they only run where that
framework is
+installed. Today that is two frameworks, [Strands
Agents](https://strandsagents.com/), an open-source
+agent SDK from AWS, and [Google ADK](https://google.github.io/adk-docs/), and
neither can be installed
+in the workspace: Strands caps `mcp`, and ADK caps `opentelemetry` and
`websockets`, below the versions
+`uv.lock` resolves. The regular test jobs therefore skip their adapter tests.
+
+This job runs once per framework, each installing its framework into the CI
image and running the
+Common AI tool tests (`providers/common/ai/tests/unit/common/ai/tools`), so a
breaking release of one
+framework does not hide the other's result. It runs on `main` only, on amd64,
when Common AI or
+common.sql code, the Common AI dependencies or `uv.lock` change, or when the
run tests everything.
+
+- Outside a canary run, every package in the image keeps its version and the
frameworks' caps on them
+ are overridden, so the adapters are tested against the same dependencies as
the rest of Airflow.
+- On a canary run, the frameworks' own dependency pins win, which is the
environment a user who
+ installs them gets.
+
+Both install the newest framework releases older than the repository's uv
`exclude-newer` window, so a
+release that breaks an adapter fails this job about that long after it ships.
The job is not a
+dependency of **Finalize Tests**, so such a release does not stop the image
cache from being pushed.
+
## Runners
Two kinds of GitHub-hosted runner are used, and which one a job gets depends
on what
diff --git a/dev/breeze/src/airflow_breeze/utils/selective_checks.py
b/dev/breeze/src/airflow_breeze/utils/selective_checks.py
index f278d044bc7..68dfad6edf7 100644
--- a/dev/breeze/src/airflow_breeze/utils/selective_checks.py
+++ b/dev/breeze/src/airflow_breeze/utils/selective_checks.py
@@ -122,6 +122,7 @@ class FileGroupForCi(Enum):
KUBERNETES_FILES = auto()
TASK_SDK_FILES = auto()
TASK_SDK_INTEGRATION_TEST_FILES = auto()
+ AGENT_FRAMEWORK_FILES = auto()
GO_SDK_FILES = auto()
JAVA_SDK_FILES = auto()
TS_SDK_FILES = auto()
@@ -482,6 +483,16 @@ CI_FILE_GROUP_MATCHES: HashableDict[FileGroupForCi] =
HashableDict(
FileGroupForCi.TASK_SDK_INTEGRATION_TEST_FILES: [
r"^task-sdk-integration-tests/.*\.py$",
],
+ FileGroupForCi.AGENT_FRAMEWORK_FILES: [
+ # The framework adapters sit on the framework-neutral tools, which
sit on the toolsets,
+ # so any code change in the provider can break them, and so can a
change to common.sql,
+ # which the SQL toolset uses, or to the locked versions the job
holds the framework to.
+ # The job's own script and workflow are ENVIRONMENT_FILES, which
run everything.
+ r"^providers/common/ai/(src|tests)/.*\.py$",
+ r"^providers/common/ai/pyproject\.toml$",
+ r"^providers/common/sql/src/.*\.py$",
+ r"^uv\.lock$",
+ ],
FileGroupForCi.GO_SDK_FILES: [
# `.md` excluded — doc-only edits do not affect the Go build or
tests, but
# everything else (go.mod, go.sum, build config) must trigger the
unit tests.
@@ -1145,6 +1156,13 @@ class SelectiveChecks:
FileGroupForCi.TASK_SDK_INTEGRATION_TEST_FILES
)
+ @cached_property
+ def run_agent_framework_tests(self) -> bool:
+ # Providers are released from main only, as for skip_providers_tests.
+ if self._default_branch != "main":
+ return False
+ return self._should_be_run(FileGroupForCi.AGENT_FRAMEWORK_FILES)
+
@cached_property
def run_go_sdk_tests(self) -> bool:
return self._should_be_run(FileGroupForCi.GO_SDK_FILES)
diff --git a/dev/breeze/tests/test_selective_checks.py
b/dev/breeze/tests/test_selective_checks.py
index f496912f6b3..533cd41a6a8 100644
--- a/dev/breeze/tests/test_selective_checks.py
+++ b/dev/breeze/tests/test_selective_checks.py
@@ -1598,6 +1598,41 @@ def assert_outputs_are_printed(expected_outputs:
dict[str, str], stderr: str):
},
id="Skip go unit and e2e tests for go-sdk ADR-only change",
),
+ pytest.param(
+
("providers/common/ai/src/airflow/providers/common/ai/toolsets/sql.py",),
+ {"run-agent-framework-tests": "true", "full-tests-needed":
"false"},
+ id="Run agent framework tests when a common.ai toolset changes",
+ ),
+ pytest.param(
+
("providers/common/ai/tests/unit/common/ai/tools/test_strands.py",),
+ {"run-agent-framework-tests": "true", "full-tests-needed":
"false"},
+ id="Run agent framework tests when a common.ai test changes",
+ ),
+ pytest.param(
+ ("providers/common/ai/pyproject.toml",),
+ {"run-agent-framework-tests": "true", "full-tests-needed":
"false"},
+ id="Run agent framework tests when common.ai dependencies change",
+ ),
+ pytest.param(
+
("providers/common/sql/src/airflow/providers/common/sql/hooks/sql.py",),
+ {"run-agent-framework-tests": "true", "full-tests-needed":
"false"},
+ id="Run agent framework tests when common.sql changes",
+ ),
+ pytest.param(
+ ("uv.lock",),
+ {"run-agent-framework-tests": "true", "full-tests-needed":
"false"},
+ id="Run agent framework tests when the lock file changes",
+ ),
+ pytest.param(
+ ("providers/common/ai/docs/frameworks/strands.rst",),
+ {"run-agent-framework-tests": "false"},
+ id="Skip agent framework tests for a common.ai docs-only change",
+ ),
+ pytest.param(
+ ("providers/openai/src/airflow/providers/openai/hooks/openai.py",),
+ {"run-agent-framework-tests": "false"},
+ id="Skip agent framework tests when another provider changes",
+ ),
pytest.param(
("airflow-e2e-tests/docker/go.yml",),
{
@@ -2544,6 +2579,11 @@ def test_expected_output_full_tests_needed(
id="No Helm tests, No providers no lint charts, should run if "
"only chart/providers changed in non-main but PROD image should be
built",
),
+ pytest.param(
+
("providers/common/ai/src/airflow/providers/common/ai/toolsets/sql.py",),
+ {"run-agent-framework-tests": "false", "skip-providers-tests":
"true"},
+ id="No agent framework tests on a release branch, which releases
no providers",
+ ),
pytest.param(
(
"airflow-core/src/airflow/cli/test.py",
diff --git a/scripts/in_container/run_agent_framework_tests.sh
b/scripts/in_container/run_agent_framework_tests.sh
new file mode 100755
index 00000000000..2053c68fdc1
--- /dev/null
+++ b/scripts/in_container/run_agent_framework_tests.sh
@@ -0,0 +1,90 @@
+#!/usr/bin/env bash
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# Runs the common.ai adapter tests for agent frameworks that cannot join the
workspace lock.
+#
+# Strands Agents caps mcp, and Google ADK caps opentelemetry and websockets,
below the versions
+# uv.lock resolves, so their adapter tests are skipped everywhere else in CI.
This installs one
+# framework into the CI image and runs the tests of the framework-neutral
tools and their adapters;
+# the other framework's tests skip themselves. One framework per invocation,
so a bad release of
+# one cannot mask the other.
+#
+# By default every package already in the image is held at its installed
version with uv's
+# --override, so the framework is tested against the same dependencies as the
rest of Airflow and
+# its caps on them are overridden. With --framework-pins the framework's own
requirements win
+# instead, which is the environment a user who installs it gets.
+#
+# The newest framework release older than the repository's uv exclude-newer
window is installed.
+set -euo pipefail
+
+TEST_PATH="providers/common/ai/tests/unit/common/ai/tools"
+
+framework="${1:-}"
+case "${framework}" in
+ strands-agents) import_check="import strands" ;;
+ google-adk) import_check="import google.adk" ;;
+ *)
+ echo "Usage: $0 <strands-agents|google-adk> [--framework-pins]" >&2
+ exit 1
+ ;;
+esac
+
+framework_pins="false"
+if [[ ${2:-} == "--framework-pins" ]]; then
+ framework_pins="true"
+elif [[ -n ${2:-} ]]; then
+ echo "Unknown argument: ${2}. The only option after the framework is
--framework-pins." >&2
+ exit 1
+fi
+
+cd "${AIRFLOW_SOURCES:-/opt/airflow}"
+
+if [[ ${framework_pins} == "true" ]]; then
+ echo "Installing ${framework} with its own dependency pins"
+ uv pip install "${framework}"
+else
+ overrides=$(mktemp)
+ before=$(mktemp)
+ after=$(mktemp)
+ trap 'rm -f "${overrides}" "${before}" "${after}"' EXIT
+ uv pip freeze | sort > "${before}"
+ # Only name==version lines: editable and local installs cannot be
expressed as an override,
+ # and the frameworks do not depend on any of them. Overriding the rest
means nothing the
+ # image ships should change; the check below is there in case something
still does.
+ grep -E '^[A-Za-z0-9_.-]+==' "${before}" > "${overrides}"
+ echo "Installing ${framework}, holding the image's $(wc -l <
"${overrides}") installed packages"
+ uv pip install --override "${overrides}" "${framework}"
+ uv pip freeze | sort > "${after}"
+ changed=$(comm -23 "${before}" "${after}")
+ if [[ -n ${changed} ]]; then
+ echo "Installing ${framework} changed packages the image already had:"
>&2
+ echo "${changed}" >&2
+ exit 1
+ fi
+fi
+
+# Log the versions that decide whether the adapter works; the framework itself
must be among them.
+uv pip freeze | grep -iE
'^(strands-agents|google-adk|mcp|opentelemetry-(api|sdk)|websockets|google-genai)=='
\
+ || { echo "${framework} is not installed after uv pip install" >&2; exit
1; }
+
+# The adapter tests skip themselves when their framework is missing, so a
broken install would
+# otherwise pass as green.
+python -c "${import_check}"
+
+# --skip-db-tests: the job runs with backend "none", which has no database to
set up.
+pytest "${TEST_PATH}" --skip-db-tests -p no:cacheprovider --color=yes -ra
