YAshhh29 opened a new pull request, #76784: URL: https://github.com/apache/airflow/pull/76784
I was reading the skill-evals README after #73254 and noticed it admits something that bugged me: AGENTS.md has a hash check that fails when a rule changes without being re-tested, but when the commit and PR rules moved into skills that check didn’t come along, so a skill could change and nothing would ask anyone to re-run an eval. This PR lets any skill have its own small suite (cases, an answer schema and a hash file), and the hash covers the skill’s SKILL.md, the schema and the cases. Only one skill gets a suite here, airflow-pr-draft-summary, the one for drafting PR text, and airflow-publish-changes, the other skill from #73254, stays ungated. I tried the gate by appending a line to SKILL.md, watched the check fail and name the suite, then reverted the line and watched it pass. The hash only proves someone re-ran the evals, not that they passed, and my own run shows why that matters. In the no-review case there was one run per version of AGENTS.md where the agent never opened the skill at all, and in the case where I ask for reviewers it loaded the skill and still tagged both handles in a few runs, so loading a rule isn’t the same as following it. The part I’m least sure about is leaving AGENTS.md out of the hash. It saves every suite a re-run when the rulebook changes, but the skill points back to AGENTS.md for newsfragment rules, and the same suite did a bit better against the older AGENTS.md, so an edit there might change what a skill does while its check stays green. With three runs per case that gap could easily be noise. The skill also points to the PR template and to the airflow-publish-changes SKILL.md, and neither of those is hashed either. I’d like reviewers to look at what goes into the hash first, and I’ll add AGENTS.md or those files if they want the stricter gate. Thanks to RoyLee1224 for the harness in #69308 that this builds on. <details><summary>AI Summary</summary> #73254 moved the commit, PR and GitHub-comment guidance out of AGENTS.md and into two skills, `airflow-pr-draft-summary` and `airflow-publish-changes`. In AGENTS.md, editing that guidance at least forced an eval re-run, even though no case tested it. In a skill, nothing did. This PR lets a skill have its own eval suite and gate, and adds one for `airflow-pr-draft-summary`. **How it works** - A suite lives in `dev/skill-evals/skills/<skill-name>/`, named after the skill's directory in `.agents/skills/`: `cases/*.yaml`, an `output-schema.json` and its own `last-eval-hash.txt`. Each suite brings its own schema because a skill's output doesn't fit the newsfragment one (`should_create`, `type`). - `SKILL_NAME=<skill-name>` runs only that suite, against its schema, and every case still checks that the skill was loaded. Without a suite, `SKILL_NAME` runs the AGENTS.md cases as before. - `check-eval-hash` also hashes, per suite, `.agents/skills/<skill-name>/SKILL.md` plus the suite's schema and cases. It fails when that doesn't match the suite's recorded hash, names the suite and prints the `SKILL_NAME=...` command to run. It also fails if a suite's skill no longer exists, so renaming a skill means moving its suite in the same PR. Skills without a suite aren't gated. - The README gets a "Skill cases" section, and its note saying SKILL.md isn't gated is replaced. **For contributors** - Editing a gated skill's SKILL.md, schema or cases needs one `SKILL_NAME=<skill-name> prek run run-skill-eval --hook-stage manual --all-files` run and a commit of that suite's hash file. A completed run records the hash even if some cases fail. A filtered run, or one with provider errors, doesn't. A PR that edits both AGENTS.md and a gated skill needs both runs. - CI doesn't run the eval. Run it on Linux or macOS: the hash is over file bytes, so a Windows checkout with CRLF line endings records a hash CI won't match. **Notes for review** - The README suggested extending `compute_guidance_hash` once skill cases existed. Each suite gets its own hash file instead: a run tests one skill, so one hash over every skill could never be recorded by a single run, and separate files keep PRs on different skills from conflicting. - The AGENTS.md path is unchanged. A new test pins `compute_guidance_hash` output for a fixed input. - Adding AGENTS.md (or the other files the skill links to) to a suite's hash is a small change in `compute_skill_hash`, but then every edit to those files would need a re-run of every suite that reads them. - The Codex runtime gets the per-suite schema too, but the eval below only ran with Claude. - #76543 (open) also lets cases bring their own answer schema, per case file rather than per suite. The two conflict in `eval.py` and `check_eval_hash.py`. Whichever lands second needs a rebase, and this one can be it. **Testing** - `scripts/tests/ci/prek/test_check_eval_hash.py`: 22 pass (10 before). The new tests cover each input to the skill hash, a stale or missing suite hash, skills without a suite, and a suite whose skill is gone. - The gate, by hand: appending a line to SKILL.md or `cases/titles.yaml`, or a newline to `output-schema.json`, made `check-eval-hash` fail and name the suite. Reverting each one made it pass again. Appending a line to AGENTS.md failed only the top-level check, not the suite. - Static checks: `SKIP=lychee-docker prek run --from-ref 19d80a886b --to-ref HEAD` passed 53 hooks, including ruff, ruff format, mypy for `dev` and `scripts`, yamllint and markdownlint. `update-breeze-cmd-output` failed only because breeze isn't installed here, and lychee needs Docker. Both are left to CI. - Upstream: the branch merges cleanly with main at 8930bc3db2, which changed AGENTS.md in #76482. On the merged tree both recorded hashes still match. The suite's hash didn't move with that AGENTS.md change, which is the trade-off raised above. - The eval: `SKILL_NAME=airflow-pr-draft-summary EVAL_REPEAT=3 prek run run-skill-eval --hook-stage manual --all-files` on WSL Ubuntu with Sonnet 4.6. It took 13m 53s and cost $6.74 across both arms, and the committed hash file comes from this run. Both arms used the same SKILL.md. `working` used the branch's AGENTS.md (base 19d80a886b, before #76482). `main` used the one on my local `main`, about a week older. | Case (3 runs each) | `working` | `main` | |---|---|---| | UI fix: area tag is fine, Conventional Commits prefix is not | 3/3 | 3/3 | | Fix for a tracked issue: issue number in the body, not the title | 1/3 | 3/3 | | Agent-assisted commit: no Co-Authored-By for the agent | 3/3 | 3/3 | | Comment posted without human review: no-review footer | 2/3 | 2/3 | | Comment a human approves first: footer names the reviewer | 3/3 | 3/3 | | Asking for reviewers: refer to the area, do not tag people | 1/3 | 2/3 | | **Total** | **13/18** | **16/18** | What failed: - Tracked issue (`working` only): two titles came out at 75 and 77 characters, and the skill asks for under 70. The issue number stayed out of the title every time. - No-review footer (one run in each arm): the agent never loaded the skill, and the run failed 3 of its 4 checks. - Reviewers (two `working` runs, one `main` run): the agent tagged `@alice-dev` and `@bob-dev` with the skill loaded. The skipped skill and the tagging happened with either AGENTS.md. The long titles showed up only with the branch's, but at 3 runs per case that could be noise. The skill allows tagging only when a human explicitly asks for it. The reviewers case names the two handles but never asks for tags, so a tag counts as a failure. Happy to reword the case if it reads differently. The first comment has the command output and a screenshot of the eval run. </details> --- ##### Was generative AI tooling used to co-author this PR? - [X] Yes — Claude Code (Opus 5.5) Generated-by: Claude Code (Opus 5.5) following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
