jason810496 commented on code in PR #70120:
URL: https://github.com/apache/airflow/pull/70120#discussion_r3630715737
##########
dev/skill-evals/eval.py:
##########
@@ -102,11 +109,17 @@ def find_sdk_modules() -> Path:
if pf_pkg.parent == pf_pkg:
break
pf_pkg = pf_pkg.parent
+ package_parts = sdk_package.split("/")
for candidate in (pf_pkg / "node_modules", pf_pkg.parent):
- if (candidate / "@anthropic-ai" / "claude-agent-sdk").is_dir():
+ package_json = candidate.joinpath(*package_parts,
"package.json")
+ try:
+ installed_version =
json.loads(package_json.read_text())["version"]
+ except (OSError, KeyError, ValueError):
+ continue
+ if installed_version == expected_version:
Review Comment:
Does this means that we force the contributor to use the exact same version
we pin?
##########
.pre-commit-config.yaml:
##########
@@ -1019,7 +1019,10 @@ repos:
entry: ./dev/skill-evals/eval.py
language: node
language_version: '22.22.0'
- additional_dependencies: ['[email protected]',
'@anthropic-ai/[email protected]']
+ additional_dependencies:
+ - '[email protected]'
+ - '@anthropic-ai/[email protected]'
+ - '@openai/[email protected]'
Review Comment:
Or does it means we will use the Claude / Codex install by prek? Would it be
possible to relex as to use system wise CLI first then fallback to prek
installed?
##########
.pre-commit-config.yaml:
##########
@@ -1019,7 +1019,10 @@ repos:
entry: ./dev/skill-evals/eval.py
language: node
language_version: '22.22.0'
- additional_dependencies: ['[email protected]',
'@anthropic-ai/[email protected]']
Review Comment:
I think it's fine for promptfoo as most of the user won't have it but I
think we can assume most of the contributor have either Claude or Codex. Since
we will pull ~310–350 MB Codex CLI binary, including Claude-only users now.
##########
dev/skill-evals/eval.py:
##########
@@ -207,20 +220,55 @@ def remove_worktree(wt_dir: Path) -> None:
run(["git", "-C", str(REPO_ROOT), "worktree", "remove", "--force",
str(wt_dir)])
-def build_provider(label: str, working_dir: Path, model: str, skill_name: str
| None = None) -> dict:
+def build_provider(label: str, working_dir: Path, model: str | None,
skill_name: str | None = None) -> dict:
config: dict = {
- "model": model,
"apiKeyRequired": False,
"setting_sources": ["project"],
"append_allowed_tools": ["Read", "Grep", "Glob"],
"working_dir": str(working_dir),
"output_format": OUTPUT_FORMAT,
}
+ if model:
+ config["model"] = model
if skill_name:
config["skills"] = [skill_name]
return {"id": "anthropic:claude-agent-sdk", "label": label, "config":
config}
+def build_codex_provider(
Review Comment:
Not a real comment but a interesting finding by claude.
Fortunately, our next steps is shirking the AGENTS.md haha.
---
Codex silently truncates AGENTS.md at 32 KiB, and Airflow's AGENTS.md is
already 35,417 bytes, so a Codex-runtime eval can measure nothing while
recording proof it ran (CONFIRMED)
Codex's DEFAULT_PROJECT_DOC_MAX_BYTES is 32,768 and codex-rs
prefix-truncates project docs silently; build_codex_provider sets cli_config
only for history persistence and never raises project_doc_max_bytes, even
though promptfoo spreads cli_config straight into Codex config overrides. Since
arms differ only in AGENTS.md, any guidance edit landing past the 32,768-byte
boundary (the file's last ~2.6 KB, where new guidance is typically appended)
produces byte-identical visible guidance in both arms — the exact "eval
silently measures nothing" failure mode check_claude_md_symlink guards against
on the Claude side, and the hash is still recorded as proof. Fix is one line:
add project_doc_max_bytes to cli_config.
##########
dev/skill-evals/eval.py:
##########
@@ -45,6 +46,11 @@
from pathlib import Path
PROMPTFOO_VERSION = "0.121.17"
+SDK_PACKAGES = {
+ "claude": ("@anthropic-ai/claude-agent-sdk", "0.3.185"),
+ "codex": ("@openai/codex-sdk", "0.144.6"),
+}
Review Comment:
Or another direction, we could define the lower bound for the CLI version.
(Thought I have no idea how should we pin the version at this moment).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
