potiuk opened a new pull request, #423:
URL: https://github.com/apache/airflow-steward/pull/423
## Summary
#418 shipped the `preflight-audit` CLI with a replay mode but no fixture
exercising the full classifier end-to-end. This adds the eval-fixture pattern
the tool's README promised.
## What
- **`tests/fixtures/synthetic_workspace_sweep.json`** — 12-issue GraphQL
response, one issue per rule path:
- Rule 1 dispatch (recent human activity)
- Rule 1 yields → Rule 7 fires (skill-drove-update)
- Rule 2 dispatch-urgent (non-skill reply <24h after >7d gap)
- Rules 3–7 skip-noop (post-announce, stale, all-phases-done, awaiting
release, awaiting advisory)
- GitHub-App bot login + personal-bot-needing-override
- Fall-through dispatch + recently-closed dispatch
Each issue node carries a `_purpose` annotation so the fixture documents
its own intent.
- **`tests/test_eval_replay.py`** — drives `classify_response` against the
fixture with a pinned `now` (`2026-06-01T12:00:00Z`) and asserts:
1. The full per-decision bucket distribution by issue number
2. The same distribution under `extra_bot_logins` (one issue migrates from
`dispatch` → `skip-noop`)
3. Per-issue assertions with reason-substring matches
4. A skip-rate floor (≥30%) matching the real-world target after #416's
tuning
## Why this matters
A rule change that alters the distribution fails one of the asserts; the
diff tells the reviewer how the rule affects coverage **before they touch any
real adopter data**. The eval is deterministic (no live `gh` calls, fixed
`now`) so CI runs it in milliseconds.
This closes the tune-then-verify loop one more rung up:
- #416 used a one-off `/tmp/` script
- #418 promoted it to a CLI
- This PR locks the rule behaviour into the test suite
## Test plan
- [x] 4 new tests pass (41 total in preflight-audit)
- [x] `prek run --all-files` green
- [ ] CI: workspace pytest + lychee
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
