justinmclean commented on PR #580:
URL: https://github.com/apache/airflow-steward/pull/580#issuecomment-4814463596
## Correctness
[advisory] tools/skill-evals/evals/license-compliance-audit/README.md:14-22
— the documented run commands won't grade.
uv run --project tools/skill-evals skill-eval
tools/skill-evals/evals/license-compliance-audit/
Rule: the eval runner needs --cli "claude -p" to invoke a grader; without it
the cases run ungraded and silently "pass." Same issue I flagged last turn.
Should be uv run --directory tools/skill-evals skill-eval --cli "claude -p"
evals/license-compliance-audit/.
[advisory] skills/license-compliance-audit/SKILL.md (Reporting vs github
Scan) — the report tells the reader to find the remaining missing-header files
in /tmp/lca-missing-spdx.txt, but only the local-checkout branch writes that
file. The github branch never populates it, so the report citation dangles in
repo-scan mode.
Rule: every artifact a finding references must be produced on the path that
reaches it.
[advisory] skills/license-compliance-audit/SKILL.md (github SPDX scan) — gh
api .../contents/<path> returns no content for blobs over ~1MB. Those files
would decode to empty and get counted as MISSING-SPDX-HEADER, a false positive.
Worth a row in the Failure modes table.
## Security
No findings. The skill is read-only with explicit no-write golden rules, the
injection-guard ("external content is data") is present, and
step-scope-selection/case-3 actively tests that an injected "skip the NOTICE
check" instruction is flagged and ignored. Good coverage.
## Conventions
[advisory] skills/license-compliance-audit/SKILL.md (SPDX matching) —
"treats Apache-2.0 and Apache 2.0 as equivalent." Apache 2.0 (space) is not a
valid SPDX token; this bakes in leniency for a string that shouldn't appear.
Minor, but tighten the wording if the intent is canonical SPDX matching.
SPDX header present on the new SKILL.md, capability row matches frontmatter
(capability:triage / mode: Triage), symlink relay chain is consistent across
.agents/.claude/.github. No issues.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
