justinmclean opened a new pull request, #272:
URL: https://github.com/apache/airflow-steward/pull/272
## Summary
Advances the **Mentoring** mode from `proposed` to `experimental` and adds
the
intervention-selection eval coverage for the `pr-management-mentor` skill,
including
a new mixed-signal hand-off case.
## Changes
**Mode status**
- `docs/modes.md`: Mentoring mode `proposed | 0` → `experimental | 1`;
replaced the
"no skill yet" placeholder with a skill table and refreshed the status
narrative.
**Intervention-selection eval suite**
(`tools/skill-evals/evals/pr-management-mentor/intervention/`)
- Added the intervention-selection step fixtures (`system-prompt.md`,
`user-prompt-template.md`) implementing the ordered decision tree:
hand-off checks
(triggers 4 → 3 → 1 → 2) → maintainer-already-engaged → template matching
(1–4).
- Cases covering every branch of the tree:
- template 1 (missing reproduction), template 2 (missing version),
template 3 (convention gap), template 4 (why-pushback → hand-off)
- multiple triggers simultaneously → `ask`
- maintainer already engaged → `silent`; no trigger fires → `silent`
- out-of-scope topic → hand-off
- **new** `case-9-deprecation-decision`: an out-of-scope
deprecation/removal
decision that also carries draftable bug signals — verifies hand-off
ordering
(trigger 3) wins over template matching.
- Bug fix to the intervention `system-prompt.md`.
- `pr-management-mentor/README.md`: suite counts updated (intervention 9
cases;
29 total).
**Upstream sync**
- Merged `upstream/main` to keep the branch current.
## Testing
```bash
PYTHONPATH=tools/skill-evals/src python3 -m skill_evals.runner \
tools/skill-evals/evals/pr-management-mentor/intervention/fixtures/
```
All 9 intervention cases assemble and the model's decision matches each
`expected.json`, verified by walking the decision tree per case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
