Hi everyone,
As part of GSoC, Jason Liu and I have been developing a repeatable way to evaluate Airflow’s instructions and tools for agents. https://github.com/apache/airflow/pull/69308 added harness to run the same Airflow-specific scenarios under different configurations and compares correctness, turns, token usage, and execution traces. https://github.com/apache/airflow/pull/70120 added Codex support, and OpenCode support is in progress. The goal is to replace “this guidance seems useful” with reproducible evidence. ## Initial showcases ### 1. Newsfragment decisions https://github.com/apache/airflow/pull/67982 Unnecessary newsfragments are a recurring source of review cleanup. We converted examples from real PRs into eval cases. In an initial experiment covering four ambiguous changes, repeated three times: - With AGENTS.md: 9/12 correct - Without AGENTS.md: 1/12 correct The guidance clearly helped, but ambiguous fixes still caused failures. We later added two positive cases where a newsfragment should be created, preventing an “always omit it” strategy from passing. This demonstrates how recurring maintainer feedback can become a regression test. ### 2. MCP runtime debugging https://github.com/apache/airflow/pull/69381 We also evaluated four ways of diagnosing the real runtime bug in #39801: | Arm | Result | Turns | Tokens | | --- | --- | --- | --- | | MCP tools | Correct | 10 | 27.8k | | CLI + documented workflow | Correct | 7 | 27.4k | | Bare CLI | Correct | 28 | 54.7k | | Source inspection only | Failed at turn limit | — | — | For this scenario, runtime access was decisive. MCP and a documented CLI workflow both worked efficiently, while bare CLI access incurred substantial discovery and authentication overhead. This is an n=1 result, not a recommendation about MCP versus skills. Its purpose is to show that the eval framework can provide objective evidence about where a tool helps and what part of the workflow creates that value. ## Next step: slimming the root AGENTS.md Our next target is to evaluate whether the root AGENTS.md can be reduced without harming agent behavior. At the time of writing: - the root file contains 522 lines and 35,925 bytes; - "Commits and PRs" section contains 307 lines—about 59% of the file’s lines; - the repository has 14 AGENTS.md files: one root and 13 nested; - the root file alone exceeds Codex’s default 32,768-byte project-document limit. We plan to compare: 1. the current root file; 2. a slimmer root file; 3. the slimmer root file with procedural guidance available through skills. One prototype would keep repository-wide invariants and mandatory skill triggers in the root, while evaluating focused skills for: - code-change verification; - PR and review preparation; - publishing changes when explicitly requested. The objective is not simply to make the file shorter. We want to determine which rules must always be loaded, which belong in nested AGENTS.md files, and which workflows can be loaded on demand through skills. After establishing this comparison, we can apply the same discovery and behavioral tests to the 13 nested files. ## Feedback requested We would appreciate feedback on: - whether these examples demonstrate a useful role for Airflow-specific evals; - whether slimming the root AGENTS.md is the right next target; - which instructions must remain available on every task; - which recurring review problems should become additional eval cases. Detailed prompts, traces, results, and proposed skill contents will be kept in a tracking issue. Best regards, Jhe Chen Li GSoC Contributor Jason Liu GSoC Mentor
