This is an automated email from the ASF dual-hosted git repository. davsclaus pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/camel-performance-tests.git
commit 935c9e502d98e56ecfceb509e787265822ad0a5b Author: Claus Ibsen <[email protected]> AuthorDate: Tue Sep 15 09:42:19 2026 +0200 ai-benchmark: the harness of the local-model benchmark series against the Camel MCP server One-shot and stepwise benchmarks scored by what runs, the example and step definitions, the MCP client, the suite and server scripts, the results of the 2026-09 series, and a README for people and agents who want to repeat it with another model or another set of examples. Blog: https://camel.apache.org/blog/2026/09/camel-local-model-benchmark/ Co-Authored-By: Claude Fable 5.1 <[email protected]> Claude-Session: https://claude.ai/code/session_01Bp3538HRBPMQkb5ta9xRaj --- ai-benchmark/.gitignore | 7 + ai-benchmark/README.md | 107 ++++++++++ ai-benchmark/agent_local.py | 186 +++++++++++++++++ ai-benchmark/agent_mcp_stepwise.py | 231 +++++++++++++++++++++ ai-benchmark/examples.json | 15 ++ ai-benchmark/gen_local.py | 103 +++++++++ ai-benchmark/mcp_client.py | 76 +++++++ ai-benchmark/results-2026-09.md | 31 +++ ai-benchmark/run-suite.sh | 23 ++ ai-benchmark/run_one.sh | 44 ++++ ai-benchmark/start-server.sh | 14 ++ ai-benchmark/steps.json | 117 +++++++++++ .../stepwise-project/application.properties | 4 + ai-benchmark/stepwise-project/timer-log.camel.yaml | 15 ++ ai-benchmark/summarize_runs.py | 78 +++++++ 15 files changed, 1051 insertions(+) diff --git a/ai-benchmark/.gitignore b/ai-benchmark/.gitignore new file mode 100644 index 0000000..285a577 --- /dev/null +++ b/ai-benchmark/.gitignore @@ -0,0 +1,7 @@ +# run outputs +oneshot-*/ +stepwise/ +local/ +*.log +*.out +__pycache__/ diff --git a/ai-benchmark/README.md b/ai-benchmark/README.md new file mode 100644 index 0000000..d0eeae6 --- /dev/null +++ b/ai-benchmark/README.md @@ -0,0 +1,107 @@ +# AI benchmark: a local model builds and edits Camel integrations through the Camel MCP server + +A small harness that measures how well a model builds and edits Apache Camel integrations when its only Camel +knowledge is what the Camel MCP server (`camel mcp`) gives it: the catalog, validation, and a `camel run` loop. +It was written for the series described in the blog post +[We had a frontier AI coach a small local model through Camel](https://camel.apache.org/blog/2026/09/camel-local-model-benchmark/) +(2026-09-15), where a 22 GB local model went from 0 to 12 of 13 beginner examples over twenty runs while Camel, not +the model, was changed between runs. The results of that series are in [results-2026-09.md](results-2026-09.md). + +The harness is here so the next series can be run the same way with a different set of examples, a different model, +or a later Camel, and so anyone, human or agent, can repeat it. + +## What it measures + +Two benchmarks, both scored by what actually runs, not by reading the model's output: + +1. **One-shot** (`agent_local.py`): each example is requested with its one-line description only ("Split a batch of + items into individual messages for processing"). The model may call the catalog and validation tools, then answers + with complete files. The harness writes them, validates them with the Camel CLI, runs them with `camel run` for a few + seconds, and checks the log for the described behaviour. A failure is fed back (validator output, run errors) for up + to three rounds. Pass means: validates, starts, and shows the expected activity in the log. +2. **Stepwise** (`agent_mcp_stepwise.py`): the model edits a running integration one request at a time ("fire every + five seconds", "add a choice", "add error handling", eight requests) through the write, run, log and error tools of + the MCP server. Each step is scored on the file on disk (a regex), the properties file, the log of the running + integration after the reload, and the errors, against a reference checkpoint. + +The model is never shown the example projects or their catalog tools; the one-shot tool set is restricted to catalog +lookups and validation (`BENCH_TOOL_ALLOW` in `run-suite.sh`). + +## Prerequisites + +- Java 17+ and the Camel CLI (`camel`) on the PATH, the version you want to measure (a locally built snapshot works). +- [Ollama](https://ollama.com) with the model pulled, e.g. `ollama pull qwen3.6:35b-a3b` (any model that supports tool + calling in Ollama's chat API works; set `BENCH_MODEL`). +- Python 3.10+ (standard library only, no packages). +- macOS or Linux; the scripts are zsh (`run_one.sh`, `start-server.sh`, `run-suite.sh`). + +## Running a suite + +```bash +cd ai-benchmark +./start-server.sh # camel mcp --http on port 9090, waits until it answers +./run-suite.sh my-run # one-shot examples, then the stepwise edits; about 30 to 60 minutes on a laptop +python3 summarize_runs.py my-run # the markdown table for that run (run-suite.sh prints it too) +``` + +To compare runs, pass several tags: `python3 summarize_runs.py before after1 after2`. + +Environment variables (all optional): + +| Variable | Default | Meaning | +|---|---|---| +| `BENCH_MODEL` | `qwen3.6:35b-a3b` | the Ollama model | +| `OLLAMA_HOST` | `http://localhost:11434` | where Ollama listens | +| `MCP_URL` | `http://127.0.0.1:9090/mcp` | the Camel MCP server (Streamable HTTP) | +| `BENCH_ROUNDS` | `3` | one-shot rounds per example | +| `BENCH_TOOL_CALLS` | `10` / `12` | tool calls per round (one-shot) / per step (stepwise) | +| `BENCH_TOOL_ALLOW` | see `run-suite.sh` | regex of the tools offered in the one-shot benchmark | +| `BENCH_OUT` | `oneshot` | one-shot output directory (`run-suite.sh` sets `oneshot-<tag>`) | +| `BENCH_TAG` | model name | stepwise output directory under `stepwise/` (`run-suite.sh` sets the tag) | +| `BENCH_STEPS` | `steps.json` | the stepwise scenario | +| `BENCH_VALIDATE_PROPS`, `BENCH_VALIDATE_SOURCE` | `0` | also run `camel validate properties` and `camel validate source` on the written files (`run-suite.sh` sets both) | +| `CAMEL_MCP_JAR` | unset | run a locally built `camel-jbang-mcp` runner jar instead of the CLI plugin | + +## What comes out + +- `oneshot-<tag>/<example>/attempt<n>/`: the files the model wrote, `validate.log`, `run.log`, `probe.log`. +- `oneshot-<tag>/<example>/trace.jsonl`: every tool call and answer with timings and token counts; `result.json`: pass, + rounds, tool calls, seconds, tokens. +- `stepwise/<tag>/step<n>.trace.jsonl`, `step<n>.after.yaml`, `results.json`. +- `<tag>.log`: the wall clock of the suite. + +The useful part is the traces. After a run, read every failed attempt: what the model wrote, what the validator said, +what the runtime said. Every message that told the model what was wrong without saying what to write is a Camel +improvement waiting to be made, for a person as much as for the model; that is how the 117 findings of the first series +were found (about 15 minutes of reading per run). + +## Changing the examples + +- **One-shot:** `examples.json` is a list of `{name, prompt, expect, run_seconds, probe?}`. `prompt` is what the + model gets, `expect` is for the human reading the results, `run_seconds` how long `camel run` runs, `probe` an + optional shell command run 7 s after start (for example a `curl` against a REST example) whose output lands in + `probe.log`. The pass check looks for log activity from the route; adjust `agent_local.py` if an example needs a + specific check. The first series used the 13 beginner examples of + [camel-jbang-examples](https://github.com/apache/camel-jbang-examples); a second series should use examples the + model has not seen, such as the intermediate ones. +- **Stepwise:** `steps.json` names the project directory (`stepwise-project/`, reset to the timer-log example before + each run), the route and properties files, and the steps, each with a `request`, a `check` (`file_regex`, + `props_regex`, `log_regex`, `min_log`, `interval_min`, `max_errors`) and a `reference` of the expected files. + +## Other conditions + +- `gen_local.py`: the bare-prompt condition, no tools at all, output under `local/`. It is what the first series + measured as "0 of 13" and is the baseline any tool-assisted run should be compared with. +- The frontier condition of the first series was not scripted: a coding agent with the Camel CLI as tools built the same + examples and scored 13 of 13, with the loop turning 9 first-time passes into 13. + +## Notes for an agent running this + +- Do not run Maven and the suite at the same time, and do not rebuild `camel-jbang-mcp` while a run is in progress: + the server loads its jar lazily and a replaced jar breaks it mid-run. Restart the server after a rebuild. +- A `camel.main.durationMaxSeconds` in a generated `application.properties` overrides `--max-seconds`; `run_one.sh` has + a watchdog for that. +- A model call that takes more than ten minutes is the machine asleep, not the model; `summarize_runs.py` subtracts + those, and `run-suite.sh` runs `caffeinate` on macOS. +- Keep the examples away from the model: never offer `camel_catalog_examples` or `camel_catalog_example_file` in + `BENCH_TOOL_ALLOW` for a benchmark that uses the examples repository. diff --git a/ai-benchmark/agent_local.py b/ai-benchmark/agent_local.py new file mode 100755 index 0000000..c804329 --- /dev/null +++ b/ai-benchmark/agent_local.py @@ -0,0 +1,186 @@ +#!/usr/bin/env python3 +"""One-shot benchmark: a local model builds each example from its one-line description, with the Camel MCP +server's catalog and validation tools, in a small agent loop. + +Per example: + round 1..MAX_ROUNDS: + model may call MCP tools (catalog lookups, YAML validation) up to MAX_TOOL_CALLS times, + then answers with files in '=== FILE: name ===' format. + We write the files to <BENCH_OUT>/<name>/attempt<round>/, run validate + camel run, and if it + fails we feed the validator/run errors back and loop. +Everything (tool calls, timings, tokens) is logged to <BENCH_OUT>/<name>/trace.jsonl. +""" +import json, os, re, subprocess, sys, time, urllib.request +sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) +from mcp_client import McpClient # noqa: E402 +from gen_local import write_files, strip_think # noqa: E402 + +MODEL = os.environ.get("BENCH_MODEL", "qwen3.6:35b-a3b") +HOST = os.environ.get("OLLAMA_HOST", "http://localhost:11434";) +HERE = os.path.dirname(os.path.abspath(__file__)) +OUT = os.path.join(HERE, os.environ.get("BENCH_OUT", "oneshot")) +MAX_ROUNDS = int(os.environ.get("BENCH_ROUNDS", "3")) +MAX_TOOL_CALLS = int(os.environ.get("BENCH_TOOL_CALLS", "10")) +TOOL_RESULT_CAP = 6000 + +# Tools offered to the model: catalog lookups and validation only. No example catalog +# (that would hand the model the answer), no runtime, security, migration or dependency tools. +ALLOW = re.compile(os.environ.get("BENCH_TOOL_ALLOW", + r"^camel_(catalog_(components|component_doc|eips|eip_doc|languages|language_doc|dataformats|dataformat_doc|doc_pages|doc_page)" + r"|validate_(yaml|endpoint|route|configuration)|component_(doc|properties)|eip_doc|language_doc|dataformat_doc)")) + +SYSTEM = """You are an AI assistant helping a developer build a small Apache Camel integration that runs +with the Camel CLI (camel-jbang), written in Camel YAML DSL. + +Facts about the Camel YAML DSL you must respect: +- A route file is a top-level YAML LIST. Each entry is one of `- route:`, `- from:`, `- beans:`, `- rest:`, `- restConfiguration:`. +- A route has `from:` with `uri:` and `steps:`; each step is a map with one EIP key such as `log:`, `to:`, `setBody:`, `choice:`, `split:`, `aggregate:`, `filter:`, `circuitBreaker:`, `bean:`. +- Expressions are written as `simple: "..."`, `constant: "..."`, `groovy: "..."`, `tokenize: {token: ","}`. +- Java beans must be declared with `- beans:` using `type: "#class:<FullyQualifiedOrSimpleClassName>"` before `bean: {ref: name}`. +- Use a timer or a file consumer as the trigger so the example produces output on its own. + +Use the tools to look up component options, EIP options, a validated YAML sample of an EIP you have not used before (camel_catalog_sample), and to validate your YAML before you answer. Then answer with only the +files, each in this exact format, and nothing else: + +=== FILE: <filename> === +<content> + +Route files use the extension .camel.yaml. Add application.properties, Java beans, XSLT or input files when needed.""" + + +def ollama_chat(messages, tools): + body = json.dumps({"model": MODEL, "messages": messages, "tools": tools, "stream": False, + "options": {"temperature": 0.2, "num_ctx": 32768}}).encode() + req = urllib.request.Request(HOST + "/api/chat", data=body, headers={"Content-Type": "application/json"}) + t0 = time.time() + with urllib.request.urlopen(req, timeout=1800) as r: + data = json.load(r) + return data, time.time() - t0 + + +def to_ollama_tools(mcp_tools): + out = [] + for t in mcp_tools: + if not ALLOW.match(t["name"]): + continue + schema = t.get("inputSchema") or {"type": "object", "properties": {}} + out.append({"type": "function", "function": { + "name": t["name"], "description": (t.get("description") or "")[:300], + "parameters": schema}}) + return out + + +def run_folder(folder, secs, probe): + subprocess.run([os.path.join(HERE, "run_one.sh"), folder, str(secs), probe or ""], check=False) + v = open(os.path.join(folder, "validate.log")).read() + r = open(os.path.join(folder, "run.log")).read() + p = open(os.path.join(folder, "probe.log")).read() if os.path.exists(os.path.join(folder, "probe.log")) else "" + bad_validate = "exit=1" in v or "Validation error" in v or "no yaml files" in v + m = re.search(r"Routes startup \(total:(\d+)", r) + nroutes = int(m.group(1)) if m else 0 + # run 16 on: an ERROR line written by the route's own log step (logger = the route file) is the route's + # message, not a failure (run 15 groovy logged its rejections at ERROR); a Camel error, an exception or a + # failed delivery still counts + errs = [l for l in r.splitlines() + if re.search(r"ERROR|Exception|Caused by|Unsupported|Unknown|Failed|No bean", l) + and not (re.search(r"\.(yaml|java):\d+\s", l) and not re.search(r"Exception|Caused by|Failed delivery", l))] + activity = len(re.findall(r"\.yaml:\d+ |\.java:\d+ ", r)) > 0 or bool(p.strip()) + if not activity: + # run 17 on: a log step with its own logName (priority-logger) logs under that name, not the route file; + # any log line from a logger that is not Camel's own counts as route activity + for l in r.splitlines(): + m = re.search(r" (INFO|WARN|ERROR|DEBUG) +\d+ --- \[[^\]]*\] (\S+) +: ", l) + if m and not re.search(r"camel|Camel|MainSupport|Shutdown", m.group(2)): + activity = True + break + # run 14 on: a route that logs from its own file has loaded, even when logging.level.root=WARN (honoured since + # CAMEL-24701) hides the "Routes startup" line the count is read from + if nroutes == 0 and activity: + nroutes = 1 + ok = (not bad_validate) and nroutes > 0 and not errs and activity + return ok, v, "\n".join(errs[:15]), nroutes, activity + + +def main(): + examples = json.load(open(os.path.join(HERE, "examples.json"))) + only = sys.argv[1:] + mcp = McpClient(); mcp.initialize(); tools = to_ollama_tools(mcp.list_tools()) + log = open(os.path.join(HERE, "agent_local.log"), "a") + print(f"tools offered: {[t['function']['name'] for t in tools]}", file=log, flush=True) + for ex in examples: + n = ex["name"] + if only and n not in only: + continue + base = os.path.join(OUT, n) + if os.path.exists(os.path.join(base, "result.json")): + continue + os.makedirs(base, exist_ok=True) + trace = open(os.path.join(base, "trace.jsonl"), "w") + messages = [{"role": "system", "content": SYSTEM}, + {"role": "user", "content": f"Create a runnable Camel CLI example: {ex['prompt']}."}] + result = {"name": n, "rounds": 0, "tool_calls": 0, "ok": False, "seconds": 0, "tokens": 0} + t_start = time.time() + for rnd in range(1, MAX_ROUNDS + 1): + result["rounds"] = rnd + calls = 0 + text = "" + while True: + try: + data, secs = ollama_chat(messages, tools) + except Exception as e: + text = ""; trace.write(json.dumps({"round": rnd, "error": str(e)}) + "\n"); break + msg = data["message"]; result["tokens"] += data.get("eval_count", 0) + trace.write(json.dumps({"round": rnd, "secs": round(secs, 1), "eval": data.get("eval_count"), + "prompt_eval": data.get("prompt_eval_count"), + "tool_calls": msg.get("tool_calls"), "content": (msg.get("content") or "")[:400]}) + "\n") + trace.flush() + messages.append({"role": "assistant", "content": msg.get("content") or "", "tool_calls": msg.get("tool_calls")}) + if msg.get("tool_calls") and calls < MAX_TOOL_CALLS: + for tc in msg["tool_calls"]: + fn = tc["function"]; calls += 1; result["tool_calls"] += 1 + args = fn.get("arguments") or {} + if isinstance(args, str): + try: + args = json.loads(args) + except Exception: + args = {} + try: + out = mcp.call(fn["name"], args) + except Exception as e: + out = "ERROR: " + str(e) + out = out[:TOOL_RESULT_CAP] + trace.write(json.dumps({"round": rnd, "tool": fn["name"], "args": args, "result": out[:600]}) + "\n") + trace.flush() + messages.append({"role": "tool", "content": out, "tool_name": fn["name"]}) + continue + text = msg.get("content") or "" + break + folder = os.path.join(base, f"attempt{rnd}") + files = write_files(text, folder) + with open(os.path.join(folder, "raw.txt"), "w") as f: + f.write(text) + ok, v, errs, nroutes, activity = run_folder(folder, ex["run_seconds"], ex.get("probe")) + print(f"{n}: round{rnd} calls={calls} files={files} ok={ok} routes={nroutes} activity={activity}", file=log, flush=True) + if ok: + result["ok"] = True + break + fb = "I saved and ran your files with the Camel CLI.\n\n`camel validate yaml` output:\n" + (v.strip() or "(passed)") + if errs: + fb += "\n\nErrors from `camel run`:\n" + errs + if nroutes == 0 and not errs: + fb += "\n\nThe application started but loaded 0 routes." + if nroutes > 0 and not errs and not activity: + fb += (f"\n\nThe route loaded but produced no log output in {ex['run_seconds']} seconds; it must produce " + "output on its own (timer trigger, or create the input files it reads).") + fb += "\n\nUse the tools to check the options you are unsure about and validate the YAML, then output the complete corrected files again in the same '=== FILE: <name> ===' format." + messages.append({"role": "user", "content": fb}) + result["seconds"] = round(time.time() - t_start, 1) + json.dump(result, open(os.path.join(base, "result.json"), "w")) + json.dump(messages, open(os.path.join(base, "messages.json"), "w")) + trace.close() + print(f"{n}: DONE ok={result['ok']} rounds={result['rounds']} tool_calls={result['tool_calls']} secs={result['seconds']} tokens={result['tokens']}", file=log, flush=True) + log.close() + + +if __name__ == "__main__": + main() diff --git a/ai-benchmark/agent_mcp_stepwise.py b/ai-benchmark/agent_mcp_stepwise.py new file mode 100755 index 0000000..8d20cc9 --- /dev/null +++ b/ai-benchmark/agent_mcp_stepwise.py @@ -0,0 +1,231 @@ +#!/usr/bin/env python3 +"""Stepwise editing benchmark against camel-jbang-mcp (camel mcp --http), using the shared authoring tools +from CAMEL-24695. + +The model gets the shared authoring set (camel_catalog_doc, camel_catalog_find, camel_validate_source, +camel_get_files, camel_write_file, camel_run, camel_control, camel_get_log, camel_get_errors, +camel_eval_expression, camel_error_diagnose) plus the structured catalog tools, and a short neutral system +prompt. The harness starts the integration once with camel_run (dev mode) before step 1, then sends the +8 requests one at a time. Scoring: files on disk, log via camel_get_log, +errors via camel_get_errors, diff size, reference checkpoint after each step. + +Env: BENCH_MODEL (ollama model), MCP_URL (default http://127.0.0.1:9090/mcp), BENCH_TAG (results folder name), +BENCH_STEPS (the steps file, default steps.json), +BENCH_TOOL_CALLS (max tool calls per step, default 12). +""" +import difflib, json, os, re, sys, time, urllib.request +sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) +from mcp_client import McpClient # noqa: E402 + +MODEL = os.environ.get("BENCH_MODEL", "qwen3.6:35b-a3b") +HOST = os.environ.get("OLLAMA_HOST", "http://localhost:11434";) +MCP_URL = os.environ.get("MCP_URL", "http://127.0.0.1:9090/mcp";) +HERE = os.path.dirname(os.path.abspath(__file__)) +TAG = os.environ.get("BENCH_TAG", "mcp-" + MODEL.replace(":", "_").replace("/", "_")) +OUT = os.path.join(HERE, "stepwise", TAG) +MAX_TOOL_CALLS = int(os.environ.get("BENCH_TOOL_CALLS", "12")) +TOOL_RESULT_CAP = 6000 + +SHARED = ["camel_catalog_doc", "camel_catalog_find", "camel_catalog_sample", "camel_validate_source", "camel_get_files", "camel_write_file", + "camel_run", "camel_control", "camel_get_log", "camel_get_errors", "camel_eval_expression", "camel_error_diagnose"] +EXTRA = ["camel_catalog_components", "camel_catalog_component_doc", "camel_catalog_eips", "camel_catalog_eip_doc", + "camel_catalog_languages", "camel_catalog_language_doc", "camel_catalog_dataformats", "camel_catalog_dataformat_doc"] + +SYSTEM = ( + "You are an Apache Camel assistant helping a developer edit a running Camel integration through the Camel MCP server.\n\n" + "The project directory is {directory}. The integration {name} is already running from it in dev mode: files you write are " + "reloaded automatically.\n\n" + "Guidelines:\n" + "- To edit: camel_get_files (directory, optionally file) to read, then camel_write_file (directory, file, content) with the " + "complete file. Invalid YAML or properties is refused with errors: fix them (camel_catalog_doc has the option names) and write again.\n" + "- camel_validate_source checks content before writing; camel_eval_expression checks a simple expression; camel_get_log and " + "camel_get_errors show what the running integration did after a reload.\n" + "- camel_catalog_sample (name) shows a validated YAML sample of an EIP and where it goes (top level or step): use it " + "before writing an EIP you have not written in this file yet, and after a 'not defined in the schema' error.\n" + "- Simple: functions inside ${...}, operators between them: ${header.a} == 'b', ${body} ?: 'none'.\n" + "- Never stop, kill or restart the integration unless asked.\n" + "- If a tool call returns an error, do not repeat it with the same arguments; say what failed and what to try.\n" + "- Be concise; when done, say in one or two sentences what you changed.\n" +) + + +def ollama_chat(messages, tools): + body = json.dumps({"model": MODEL, "messages": messages, "tools": tools, "stream": False, + "options": {"temperature": 0.2, "num_ctx": 32768}}).encode() + req = urllib.request.Request(HOST + "/api/chat", data=body, headers={"Content-Type": "application/json"}) + t0 = time.time() + with urllib.request.urlopen(req, timeout=1800) as r: + data = json.load(r) + return data, time.time() - t0 + + +def to_ollama_tools(mcp_tools, names): + byname = {t["name"]: t for t in mcp_tools} + out = [] + for n in names: + t = byname.get(n) + if not t: + print(f"WARNING: tool {n} not exposed by the server", file=sys.stderr) + continue + out.append({"type": "function", "function": {"name": n, "description": t.get("description") or "", + "parameters": t.get("inputSchema") or {"type": "object", "properties": {}}}}) + return out + + +def read(path): + try: + return open(path).read() + except FileNotFoundError: + return "" + + +def jcall(mcp, name, args): + out = mcp.call(name, args) + try: + return json.loads(out), out + except Exception: + return None, out + + +def log_lines(mcp, name, limit=40): + data, raw = jcall(mcp, "camel_get_log", {"name": name, "limit": limit}) + if isinstance(data, dict): + for k in ("lines", "records", "entries"): + if k in data: + return data[k] + if isinstance(data, list): + return data + return [] + + +def score(step, project, cfg, mcp, name, before): + chk = step["check"] + route = read(os.path.join(project, cfg["route_file"])); props = read(os.path.join(project, cfg["props_file"])) + result = {"file_ok": True, "props_ok": True, "log_ok": True, "errors": 0} + if chk.get("file_regex") and not re.search(chk["file_regex"], route): + result["file_ok"] = False + if chk.get("file_regex2") and not re.search(chk["file_regex2"], route): + result["file_ok"] = False + if chk.get("file_not_regex") and re.search(chk["file_not_regex"], route): + result["file_ok"] = False + if chk.get("props_regex") and not re.search(chk["props_regex"], props): + result["props_ok"] = False + time.sleep(cfg["wait_seconds"]) + lines = log_lines(mcp, name, 40) + def lvl(l): return (l.get("level") or "").upper() + def msg(l): return l.get("message") or l.get("msg") or "" + recent = [l for l in lines if isinstance(l, dict) and lvl(l) == "INFO"] + msgs = [msg(l) for l in recent] + matched = [m for m in msgs if re.search(chk["log_regex"], m)] + if len(matched) < chk.get("min_log", 1): + result["log_ok"] = False + if chk.get("interval_min") and len(matched) >= 2: + ts = [l.get("time") or l.get("timestamp") or "" for l in recent if re.search(chk["log_regex"], msg(l))][:2] + try: + def sec(t): + t = t.split("T")[-1].split(" ")[-1]; h, m, s = t.split(":")[:3]; return int(h) * 3600 + int(m) * 60 + float(s) + if abs(sec(ts[0]) - sec(ts[1])) < chk["interval_min"]: + result["log_ok"] = False + except Exception: + pass + result["errors"] = sum(1 for l in lines if isinstance(l, dict) and lvl(l) == "ERROR") + data, _ = jcall(mcp, "camel_get_errors", {"name": name}) + if isinstance(data, dict): + result["errors"] += len(data.get("errors", []) or []) + diff = list(difflib.unified_diff(before["route"].splitlines(), route.splitlines(), lineterm="", n=0)) + diffp = list(difflib.unified_diff(before["props"].splitlines(), props.splitlines(), lineterm="", n=0)) + result["changed_lines"] = sum(1 for l in diff + diffp if (l.startswith("+") or l.startswith("-")) and not l.startswith(("+++", "---"))) + result["ok"] = result["file_ok"] and result["props_ok"] and result["log_ok"] and result["errors"] == 0 + result["log_sample"] = msgs[:5] + return result, route, props + + +def main(): + cfg = json.load(open(os.path.join(HERE, os.environ.get("BENCH_STEPS", "steps.json")))) + project = cfg["project"] + if not os.path.isabs(project): + project = os.path.join(HERE, project) + os.makedirs(OUT, exist_ok=True) + log = open(os.path.join(OUT, "run.log"), "a") + mcp = McpClient(MCP_URL); mcp.initialize(); all_tools = mcp.list_tools() + tools = to_ollama_tools(all_tools, SHARED + EXTRA) + print(f"tools: {[t['function']['name'] for t in tools]}", file=log, flush=True) + + # start the integration once, in dev mode, through the MCP server + data, raw = jcall(mcp, "camel_run", {"directory": project}) + print(f"camel_run -> {raw[:300]}", file=log, flush=True) + name = (data or {}).get("name") or os.path.basename(project) + time.sleep(6) + + messages = [{"role": "system", "content": SYSTEM.replace("{directory}", project).replace("{name}", name)}] + results = [] + try: + for step in cfg["steps"]: + sid = step["id"] + before = {"route": read(os.path.join(project, cfg["route_file"])), "props": read(os.path.join(project, cfg["props_file"]))} + trace = open(os.path.join(OUT, f"step{sid}.trace.jsonl"), "w") + messages.append({"role": "user", "content": step["request"]}) + calls = 0; tokens = 0; t0 = time.time(); writes = 0; refused = 0; answer = "" + while True: + try: + data, secs = ollama_chat(messages, tools) + except Exception as e: + trace.write(json.dumps({"error": str(e)}) + "\n"); break + msg = data["message"]; tokens += data.get("eval_count", 0) + trace.write(json.dumps({"secs": round(secs, 1), "eval": data.get("eval_count"), "prompt_eval": data.get("prompt_eval_count"), + "tool_calls": msg.get("tool_calls"), "content": (msg.get("content") or "")[:500]}) + "\n"); trace.flush() + messages.append({"role": "assistant", "content": msg.get("content") or "", "tool_calls": msg.get("tool_calls")}) + if msg.get("tool_calls") and calls < MAX_TOOL_CALLS: + for tc in msg["tool_calls"]: + fn = tc["function"]; calls += 1 + args = fn.get("arguments") or {} + if isinstance(args, str): + try: + args = json.loads(args) + except Exception: + args = {} + args = dict(args) + if fn["name"] in ("camel_get_files", "camel_write_file", "camel_validate_source", "camel_run") and "directory" not in args: + args["directory"] = project + if fn["name"] in ("camel_get_log", "camel_get_errors", "camel_control", "camel_eval_expression") and "name" not in args: + args["name"] = name + if fn["name"] == "camel_write_file": + writes += 1 + try: + out = mcp.call(fn["name"], args) + except Exception as e: + out = "ERROR: " + str(e) + if fn["name"] == "camel_write_file" and ('"invalid"' in out or out.startswith("ERROR")): + refused += 1 + out = out[:TOOL_RESULT_CAP] + trace.write(json.dumps({"tool": fn["name"], "args": {k: (v if k != "content" else v[:1500]) for k, v in args.items()}, "result": out[:800]}) + "\n"); trace.flush() + messages.append({"role": "tool", "content": out, "tool_name": fn["name"]}) + continue + answer = msg.get("content") or "" + break + res, route_after, props_after = score(step, project, cfg, mcp, name, before) + res.update({"step": sid, "request": step["request"], "tool_calls": calls, "writes": writes, "refused_writes": refused, + "seconds": round(time.time() - t0, 1), "tokens": tokens, "answer": answer[:400]}) + results.append(res) + with open(os.path.join(OUT, f"step{sid}.after.yaml"), "w") as f: + f.write(route_after) + print(f"step{sid}: ok={res['ok']} file={res['file_ok']} props={res['props_ok']} log={res['log_ok']} errors={res['errors']} " + f"changed_lines={res['changed_lines']} calls={calls} writes={writes} refused={refused} secs={res['seconds']} tokens={tokens}", file=log, flush=True) + trace.close() + if step.get("reference"): + for fname, content in step["reference"].items(): + with open(os.path.join(project, fname), "w") as f: + f.write(content) + time.sleep(4) + if not res["ok"]: + messages.append({"role": "user", "content": "I fixed that step myself; the files now contain the correct version. Continue with the next request."}) + json.dump(results, open(os.path.join(OUT, "results.json"), "w"), indent=1) + finally: + data, raw = jcall(mcp, "camel_control", {"name": name, "action": "stop"}) + print(f"camel_control stop -> {raw[:200]}", file=log, flush=True) + print(f"DONE passed={sum(1 for r in results if r['ok'])}/{len(results)}", file=log, flush=True) + log.close() + + +if __name__ == "__main__": + main() diff --git a/ai-benchmark/examples.json b/ai-benchmark/examples.json new file mode 100644 index 0000000..ae62d26 --- /dev/null +++ b/ai-benchmark/examples.json @@ -0,0 +1,15 @@ +[ + {"name": "timer-log", "prompt": "Simple timer that logs a hello message every second", "expect": "a log line with a hello message roughly once per second", "run_seconds": 8}, + {"name": "cron-log", "prompt": "Scheduled task that logs every 5 seconds", "expect": "a log line every 5 seconds driven by a cron schedule", "run_seconds": 12}, + {"name": "rest-api", "prompt": "REST API with hello endpoints", "expect": "an HTTP server starts and GET on a hello endpoint returns a greeting", "run_seconds": 12, "probe": "curl -s -m 3 http://localhost:8080/api/hello; echo; curl -s -m 3 http://localhost:8080/hello; echo"}, + {"name": "routes", "prompt": "Define routes in YAML with Java beans", "expect": "a YAML route calls a Java bean and logs the bean's output", "run_seconds": 8}, + {"name": "tui-hello-world", "prompt": "Say hello via TUI Send Message (F2) or CLI", "expect": "a route that can receive a message sent from the CLI or TUI and logs a hello for it", "run_seconds": 8}, + {"name": "content-based-router", "prompt": "Route messages to different destinations based on message content using the Choice EIP", "expect": "messages with different content are logged to different branches via choice/when/otherwise", "run_seconds": 8}, + {"name": "splitter", "prompt": "Split a batch of items into individual messages for processing", "expect": "one batch message is split and each item is logged separately", "run_seconds": 8}, + {"name": "aggregator", "prompt": "Collect individual messages into a batch using the Aggregator EIP", "expect": "several individual messages are aggregated and one batch log line appears", "run_seconds": 10}, + {"name": "circuit-breaker", "prompt": "Use the circuit breaker EIP for fault tolerance", "expect": "a circuitBreaker EIP wraps a failing step and the route keeps running", "run_seconds": 10}, + {"name": "groovy", "prompt": "Use Groovy with extra dependencies and content-based routing", "expect": "a Groovy expression using a third-party dependency drives a choice and the result is logged", "run_seconds": 10}, + {"name": "xslt", "prompt": "Basic XML transformation using XSLT style sheets", "expect": "an XML input is transformed by an XSLT stylesheet and the result is logged", "run_seconds": 8}, + {"name": "message-size", "prompt": "Track message body and header sizes per endpoint", "expect": "routes produce messages of different sizes to different endpoints so sizes can be tracked", "run_seconds": 8}, + {"name": "memory-leak", "prompt": "Simulates a memory leak for testing JFR Old Object Sample diagnostics", "expect": "a route keeps allocating and retaining memory and logs growth", "run_seconds": 8} +] diff --git a/ai-benchmark/gen_local.py b/ai-benchmark/gen_local.py new file mode 100755 index 0000000..8d6edb5 --- /dev/null +++ b/ai-benchmark/gen_local.py @@ -0,0 +1,103 @@ +#!/usr/bin/env python3 +"""Generate Camel examples from one-line prompts with a local Ollama model. + +Condition B1: bare prompt, no tools. Writes files under local/<name>/attempt1/. +Condition B2 is applied later by bench_run.py: if attempt1 fails validation, the +validator output is fed back once and the result goes to local/<name>/attempt2/. +""" +import json, os, re, sys, time, urllib.request + +MODEL = os.environ.get("BENCH_MODEL", "qwen3.6:35b-a3b") +HOST = os.environ.get("OLLAMA_HOST", "http://localhost:11434";) +HERE = os.path.dirname(os.path.abspath(__file__)) +OUT = os.path.join(HERE, "local") + +SYSTEM = """You are an Apache Camel 4.23 expert. The user wants a small runnable example for the Camel CLI (camel-jbang). +Write it in Camel YAML DSL. Use the file extension .camel.yaml for routes. If the example needs extra files +(application.properties, a Java bean, an XSLT stylesheet, an input file), include them too. +Do not use Maven, Spring Boot or Quarkus. Do not explain. Output only files, each in this exact format: + +=== FILE: <filename> === +<file content> + +Output nothing before the first '=== FILE:' line and nothing after the last file.""" + + +def chat(messages): + body = json.dumps({"model": MODEL, "messages": messages, "stream": False, + "options": {"temperature": 0.2, "num_ctx": 16384}}).encode() + req = urllib.request.Request(HOST + "/api/chat", data=body, headers={"Content-Type": "application/json"}) + t0 = time.time() + with urllib.request.urlopen(req, timeout=900) as r: + data = json.load(r) + return data["message"]["content"], time.time() - t0, data + + +def strip_think(text): + return re.sub(r"<think>.*?</think>", "", text, flags=re.S).strip() + + +def write_files(text, folder): + os.makedirs(folder, exist_ok=True) + text = strip_think(text) + parts = re.split(r"^=== FILE: (.+?) ===\s*$", text, flags=re.M) + # run 11 on: a file block written inside a ``` fence ends at the closing fence, so an explanation the model + # appends after the last fence no longer lands in that file (runs 1-10 folded it into the file, and the + # validator then named it, which cost a round or the example) + fenced = os.environ.get("BENCH_FENCE_AWARE", "1") == "1" + def unfence(content): + lines = content.strip("\n").split("\n") + if fenced and lines and lines[0].strip().startswith("```"): + body = [] + for l in lines[1:]: + if l.strip().startswith("```"): + break + body.append(l) + return "\n".join(body) + return "\n".join(l for l in lines if not l.strip().startswith("```")) + parts = [parts[0]] + [unfence(p) if i % 2 == 0 else p for i, p in enumerate(parts[1:], 1)] + written = [] + if len(parts) < 3: + # model ignored the format; dump as a single yaml file + with open(os.path.join(folder, "route.camel.yaml"), "w") as f: + f.write(text.strip() + "\n") + return ["route.camel.yaml (fallback)"] + for i in range(1, len(parts), 2): + name = os.path.basename(parts[i].strip()) + content = parts[i + 1].strip("\n") + "\n" + with open(os.path.join(folder, name), "w") as f: + f.write(content) + written.append(name) + return written + + +def main(): + examples = json.load(open(os.path.join(HERE, "examples.json"))) + only = sys.argv[1:] + log = open(os.path.join(HERE, "gen_local.log"), "a") + for ex in examples: + if only and ex["name"] not in only: + continue + folder = os.path.join(OUT, ex["name"], "attempt1") + if os.path.exists(os.path.join(folder, "raw.txt")): + continue + user = f"Create a runnable Camel CLI example: {ex['prompt']}." + msgs = [{"role": "system", "content": SYSTEM}, {"role": "user", "content": user}] + try: + text, secs, data = chat(msgs) + except Exception as e: + print(f"{ex['name']}: ERROR {e}", file=log, flush=True) + continue + os.makedirs(folder, exist_ok=True) + with open(os.path.join(folder, "raw.txt"), "w") as f: + f.write(text) + with open(os.path.join(folder, "messages.json"), "w") as f: + json.dump(msgs, f) + files = write_files(text, folder) + ev = data.get("eval_count", 0); pe = data.get("prompt_eval_count", 0) + print(f"{ex['name']}: {secs:.1f}s prompt={pe} gen={ev} files={files}", file=log, flush=True) + log.close() + + +if __name__ == "__main__": + main() diff --git a/ai-benchmark/mcp_client.py b/ai-benchmark/mcp_client.py new file mode 100755 index 0000000..7de899f --- /dev/null +++ b/ai-benchmark/mcp_client.py @@ -0,0 +1,76 @@ +#!/usr/bin/env python3 +"""Minimal MCP Streamable HTTP client (protocol 2025-03-26) for camel-jbang-mcp.""" +import json, os, urllib.request, itertools + + +class McpClient: + def __init__(self, url=os.environ.get("MCP_URL", "http://localhost:9090/mcp";)): + self.url = url + self.session = None + self._ids = itertools.count(1) + self.tools = [] + + def _post(self, payload, expect_result=True): + data = json.dumps(payload).encode() + headers = {"Content-Type": "application/json", "Accept": "application/json, text/event-stream"} + if self.session: + headers["Mcp-Session-Id"] = self.session + req = urllib.request.Request(self.url, data=data, headers=headers) + with urllib.request.urlopen(req, timeout=120) as r: + sid = r.headers.get("Mcp-Session-Id") + if sid: + self.session = sid + ctype = r.headers.get("Content-Type", "") + body = r.read().decode() + if not expect_result: + return None + if "text/event-stream" in ctype: + # take the last JSON data: line + msgs = [l[5:].strip() for l in body.splitlines() if l.startswith("data:")] + for m in reversed(msgs): + try: + obj = json.loads(m) + except json.JSONDecodeError: + continue + if "result" in obj or "error" in obj: + return obj + raise RuntimeError("no JSON-RPC result in SSE body: " + body[:300]) + return json.loads(body) if body.strip() else None + + def initialize(self): + res = self._post({"jsonrpc": "2.0", "id": next(self._ids), "method": "initialize", + "params": {"protocolVersion": "2025-03-26", "capabilities": {}, + "clientInfo": {"name": "bench", "version": "0"}}}) + self._post({"jsonrpc": "2.0", "method": "notifications/initialized"}, expect_result=False) + return res + + def list_tools(self): + tools, cursor = [], None + while True: + params = {"cursor": cursor} if cursor else {} + res = self._post({"jsonrpc": "2.0", "id": next(self._ids), "method": "tools/list", "params": params}) + tools.extend(res["result"]["tools"]) + cursor = res["result"].get("nextCursor") + if not cursor: + break + self.tools = tools + return self.tools + + def call(self, name, arguments): + res = self._post({"jsonrpc": "2.0", "id": next(self._ids), "method": "tools/call", + "params": {"name": name, "arguments": arguments or {}}}) + if "error" in res: + return "ERROR: " + json.dumps(res["error"])[:2000] + content = res["result"].get("content", []) + text = "\n".join(c.get("text", "") for c in content if c.get("type") == "text") + if res["result"].get("isError"): + text = "ERROR: " + text + return text + + +if __name__ == "__main__": + c = McpClient() + print(json.dumps(c.initialize().get("result", {}).get("serverInfo"))) + for t in c.list_tools(): + print(f"{t['name']:45} {t.get('description','')[:110].replace(chr(10),' ')}") + print("total tools:", len(c.tools)) diff --git a/ai-benchmark/results-2026-09.md b/ai-benchmark/results-2026-09.md new file mode 100644 index 0000000..10d90d3 --- /dev/null +++ b/ai-benchmark/results-2026-09.md @@ -0,0 +1,31 @@ +# Results of the first series, 2026-09-12 to 2026-09-13 + +Twenty runs of the same 13 one-shot examples and 8 stepwise edits with the same local model (`qwen3.6:35b-a3b` via Ollama +on an Apple M4 Pro, 64 GB), against the Camel MCP server. What changed between runs was Camel: after each run every failed +attempt was read, the unclear message became a hint or a check, and the fix was merged before the next run. The story is +in the blog post: https://camel.apache.org/blog/2026/09/camel-local-model-benchmark/ + +Column "before" is the Camel main branch of 2026-09-12 with the shared authoring tools (CAMEL-24695) only. Runs 1 to 20 +add the fixes as they were made; run 2 was stopped and is not listed. Everything measured here is in Camel 4.23. + +| | before | run1 | run3 | run4 | run5 | run6 | run7 | run8 | run9 | run10 | run11 | run12 | run13 | run14 | run15 | run16 | run17 | run18 | run19 | run20 | +|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---| +| One-shot passes of 13 | 10 | 7 | 10 | 10 | 9 | 10 | 8 | 10 | 13 | 10 | 9 | 11 | 10 | 10 | 10 | 10 | 11 | 10 | 12 | 12 | +| One-shot first-round passes | 5 | 1 | 2 | 5 | 4 | 4 | 4 | 4 | 3 | 2 | 4 | 3 | 4 | 5 | 5 | 4 | 5 | 5 | 9 | 7 | +| One-shot rounds / tool calls | 28 / 84 | 33 / 82 | 29 / 79 | 26 / 70 | 28 / 51 | 28 / 70 | 28 / 58 | 29 / 83 | 26 / 71 | 32 / 65 | 27 / 74 | 28 / 70 | 26 / 65 | 24 / 60 | 25 / 53 | 25 / 59 | 24 / 66 | 24 / 60 | 19 / 50 | 21 / 52 | +| One-shot tokens | 169,596 | 103,908 | 63,030 | 75,355 | 72,695 | 110,119 | 83,348 | 132,131 | 42,482 | 80,185 | 81,556 | 106,498 | 52,370 | 104,892 | 53,839 | 52,828 | 47,067 | 97,521 | 60,105 | 61,173 | +| Stepwise passes of 8 (strict / lenient) | 5 / 5 | 5 / 6 | 8 / 8 | 7 / 8 | 8 / 8 | 7 / 8 | 8 / 8 | 7 / 7 | 8 / 8 | 8 / 8 | 8 / 8 | 8 / 8 | 7 / 7 | 8 / 8 | 8 / 8 | 8 / 8 | 8 / 8 | 8 / 8 | 8 / 8 | 7 / 8 | +| Stepwise tool calls / refused writes | 34 / 9 | 34 / 11 | 31 / 2 | 20 / 2 | 31 / 4 | 24 / 3 | 21 / 2 | 27 / 4 | 22 / 2 | 19 / 3 | 40 / 9 | 30 / 4 | 22 / 3 | 15 / 2 | 19 / 2 | 23 / 2 | 19 / 2 | 26 / 2 | 20 / 2 | 24 / 2 | +| Stepwise tokens | 41,264 | 19,773 | 7,896 | 6,567 | 9,975 | 10,778 | 9,901 | 11,390 | 8,201 | 12,843 | 15,424 | 11,421 | 8,755 | 9,277 | 6,370 | 7,802 | 9,365 | 11,834 | 7,629 | 9,975 | +| Suite wall clock (sleep stalls removed) | 72 min (56 + 16) | 45 min (36 + 9) | 31 min (25 + 6) | 33 min (28 + 4) | 35 min (28 + 6) | 45 min (39 + 6) | 38 min (31 + 6) | 52 min (46 + 6), 47 min asleep | 116 min (110 + 5), 46 min asleep | 39 min (33 + 6) | 38 min (29 + 8) | 44 min (37 + 6) | 29 min (24 + 5) | 42 min (37 + 5) | 27 min (22 + 4) | 27 min (22 + 5) | 25 min (20 + 5) | 41 min (34 + 6) | 28 min (23 + 5) | 29 min (23 + 6) | +| One-shot failures | rest-api, aggregator, memory-leak | rest-api, routes, tui-hello-world, aggregator, circuit-breaker, groovy | content-based-router, aggregator, memory-leak | aggregator, xslt, memory-leak | splitter, aggregator, xslt, message-size | routes, groovy, memory-leak | routes, splitter, aggregator, groovy, memory-leak | circuit-breaker, xslt, memory-leak | none | routes, groovy, xslt | routes, aggregator, xslt, message-size | xslt, memory-leak | content-based-router, aggreg [...] +| Stepwise failures | 6, 7, 8 | 2, 4, 6 | none | 2 | none | 2 | none | 7 | none | none | none | none | 4 | none | none | none | none | none | none | 2 | + +Reading the table: + +- One-shot passes went from 7 (strict) to 12 of 13; first-round passes from 5 to 9 (run 19); tokens from 170k to about 60k. +- The stepwise edits reached 8 of 8 from run 3 on and stayed there. +- From run 14 the remaining one-shot failures were the model's (thinking spirals of 20k to 27k tokens, answering with + pasted files instead of the write tool after a correct hint), not Camel's; the series stopped at run 20 for that reason. +- The wall clock removes the time the laptop was asleep (a model call longer than 600 s); `caffeinate` in `run-suite.sh` + keeps it awake now. diff --git a/ai-benchmark/run-suite.sh b/ai-benchmark/run-suite.sh new file mode 100755 index 0000000..5ce2551 --- /dev/null +++ b/ai-benchmark/run-suite.sh @@ -0,0 +1,23 @@ +#!/bin/zsh +# Runs one full suite: the one-shot examples, then the stepwise edits. Usage: run-suite.sh <tag> +# Results: oneshot-<tag>/, stepwise/<tag>/, <tag>.log (wall clock). Summarise with: summarize_runs.py <tag> +set -u +TAG="${1:?usage: run-suite.sh <tag>}" +cd "$(dirname "$0")" +export MCP_URL="${MCP_URL:-http://127.0.0.1:9090/mcp}"; +export BENCH_VALIDATE_PROPS=1 +export BENCH_VALIDATE_SOURCE=1 +# the one-shot model gets catalog lookups and validation only: no example catalog (that would hand it the answer), no runtime tools +export BENCH_TOOL_ALLOW="${BENCH_TOOL_ALLOW:-^camel_(catalog_(components|component_doc|eips|eip_doc|languages|language_doc|dataformats|dataformat_doc|docs|doc|find|sample)|validate_(yaml_dsl|route|source)|component_properties|configuration_validate|error_diagnose|eval_expression)$}" +command -v caffeinate > /dev/null && caffeinate -i -s -w $$ & # macOS: keep the machine awake for the hour +echo "[$TAG] one-shot start $(date +%T)" | tee -a "$TAG.log" +BENCH_OUT="oneshot-$TAG" python3 agent_local.py > "oneshot-$TAG.out" 2>&1 +echo "[$TAG] one-shot done $(date +%T)" | tee -a "$TAG.log" +# the stepwise project starts from the timer-log example every time +git checkout -q -- stepwise-project 2>/dev/null || true +echo "[$TAG] stepwise start $(date +%T)" | tee -a "$TAG.log" +BENCH_TAG="$TAG" python3 agent_mcp_stepwise.py > "stepwise-$TAG.out" 2>&1 +echo "[$TAG] stepwise done $(date +%T)" | tee -a "$TAG.log" +git checkout -q -- stepwise-project 2>/dev/null || true +echo "[$TAG] DONE" | tee -a "$TAG.log" +python3 summarize_runs.py "$TAG" diff --git a/ai-benchmark/run_one.sh b/ai-benchmark/run_one.sh new file mode 100755 index 0000000..1eff64c --- /dev/null +++ b/ai-benchmark/run_one.sh @@ -0,0 +1,44 @@ +#!/bin/zsh +# Usage: run_one.sh <folder> <seconds> [probe command] +# Validates every *.camel.yaml in <folder> with `camel validate yaml`, then runs the +# folder for <seconds> and captures the log. Optional probe runs 6s after start. +set -u +folder="$1"; secs="$2"; probe="${3:-}" +cd "$folder" || exit 2 +: > validate.log; : > run.log; : > probe.log +files=(*.camel.yaml(N) *.yaml(N)) +if (( ${#files} == 0 )); then echo "no yaml files" > validate.log; fi +for f in ${(u)files}; do + echo "### $f" >> validate.log + camel validate yaml "$f" >> validate.log 2>&1 + echo "exit=$?" >> validate.log +done +# properties files: camel.* keys against the catalog (camel validate properties, CAMEL-24698) +if [[ "${BENCH_VALIDATE_SOURCE:-0}" == "1" ]]; then + for p in *.java(N) *.xsl(N) *.xslt(N) *.xml(N); do + echo "### $p" >> validate.log + camel validate source "$p" >> validate.log 2>&1 + echo "exit=$?" >> validate.log + done +fi +if [[ "${BENCH_VALIDATE_PROPS:-0}" == "1" ]]; then + for p in *.properties(N); do + echo "### $p" >> validate.log + camel validate properties "$p" >> validate.log 2>&1 + echo "exit=$?" >> validate.log + done +fi +# run everything in the folder (yaml, java, properties are picked up by camel run *) +( camel run * --max-seconds="$secs" --logging-color=false > run.log 2>&1 ) & +pid=$! +# watchdog: a camel.main.durationMaxSeconds in the example's own application.properties overrides --max-seconds +# (run 9 memory-leak ran for an hour); kill the run after the expected time plus a grace period +( sleep $((secs + 45)); pkill -TERM -f -- "--max-seconds=$secs --logging-color=false" 2>/dev/null; kill -TERM $pid 2>/dev/null ) & +watchdog=$! +if [[ -n "$probe" ]]; then + sleep 7 + eval "$probe" > probe.log 2>&1 +fi +wait $pid +echo "run-exit=$?" >> run.log +kill $watchdog 2>/dev/null diff --git a/ai-benchmark/start-server.sh b/ai-benchmark/start-server.sh new file mode 100755 index 0000000..9bc15de --- /dev/null +++ b/ai-benchmark/start-server.sh @@ -0,0 +1,14 @@ +#!/bin/zsh +# Starts the Camel MCP server with HTTP transport on port 9090 (camel mcp --http) and waits until it answers. +# Set CAMEL_MCP_JAR to run a locally built camel-jbang-mcp runner jar instead of the CLI plugin. +cd "$(dirname "$0")" +PORT="${MCP_PORT:-9090}" +pkill -f "camel-jbang-mcp.*runner.jar" 2>/dev/null; pkill -f "camel mcp --http" 2>/dev/null; sleep 1 +if [[ -n "${CAMEL_MCP_JAR:-}" ]]; then + nohup sh -c "tail -f /dev/null | java -Dquarkus.http.host-enabled=true -Dquarkus.http.host=0.0.0.0 -Dquarkus.http.port=$PORT -Dquarkus.log.level=WARN -jar $CAMEL_MCP_JAR" > mcp-server.log 2>&1 & +else + nohup sh -c "tail -f /dev/null | camel mcp --http --port=$PORT --log-level=WARN" > mcp-server.log 2>&1 & +fi +export MCP_URL="http://127.0.0.1:$PORT/mcp"; +for i in $(seq 1 45); do sleep 2; python3 mcp_client.py > /dev/null 2>&1 && { echo "server up after $((i*2))s at $MCP_URL"; python3 mcp_client.py | tail -1; exit 0; }; done +echo "server did not come up; see mcp-server.log"; tail -20 mcp-server.log; exit 1 diff --git a/ai-benchmark/steps.json b/ai-benchmark/steps.json new file mode 100644 index 0000000..3bfa65a --- /dev/null +++ b/ai-benchmark/steps.json @@ -0,0 +1,117 @@ +{ + "project": "stepwise-project", + "route_file": "timer-log.camel.yaml", + "props_file": "application.properties", + "wait_seconds": 13, + "steps": [ + { + "id": 1, + "request": "Change the timer so it fires every 5 seconds instead of every second.", + "check": { + "props_regex": "timer\\.period\\s*=\\s*5000|period.*5000", + "log_regex": "Hello Camel", + "min_log": 1, + "interval_min": 4.5 + }, + "reference": { + "timer-log.camel.yaml": "- route:\n id: timer-log\n from:\n uri: timer\n parameters:\n timerName: tick\n period: \"{{timer.period}}\"\n steps:\n - setBody:\n expression:\n simple:\n expression: \"{{greeting.message}}\"\n - log:\n message: \"${body}\"\n", + "application.properties": "timer.period=5000\ngreeting.message=Hello Camel!\n" + } + }, + { + "id": 2, + "request": "Set the body to a random number between 0 and 40 instead of the greeting message.", + "check": { + "file_regex": "random", + "log_regex": "^\\d{1,2}$", + "min_log": 1 + }, + "reference": { + "timer-log.camel.yaml": "- route:\n id: timer-log\n from:\n uri: timer\n parameters:\n timerName: tick\n period: \"{{timer.period}}\"\n steps:\n - setBody:\n expression:\n simple:\n expression: \"${random(0,40)}\"\n - log:\n message: \"${body}\"\n", + "application.properties": "timer.period=5000\ngreeting.message=Hello Camel!\n" + } + }, + { + "id": 3, + "request": "Add a choice: when the body is 30 or more, log \"hot: <the body>\", otherwise log \"normal: <the body>\".", + "check": { + "file_regex": "choice", + "log_regex": "^(hot|normal): \\d+", + "min_log": 1 + }, + "reference": { + "timer-log.camel.yaml": "- route:\n id: timer-log\n from:\n uri: timer\n parameters:\n timerName: tick\n period: \"{{timer.period}}\"\n steps:\n - setBody:\n expression:\n simple:\n expression: \"${random(0,40)}\"\n - choice:\n when:\n - simple: \"${body} >= 30\"\n steps:\n - log:\n message: \"hot: ${body}\"\n [...] + "application.properties": "timer.period=5000\ngreeting.message=Hello Camel!\n" + } + }, + { + "id": 4, + "request": "Instead of logging the hot ones, send them to the seda:hot queue. Keep logging the normal ones.", + "check": { + "file_regex": "seda:hot", + "file_not_regex": "hot: \\$", + "log_regex": "^normal: \\d+", + "min_log": 0 + }, + "reference": { + "timer-log.camel.yaml": "- route:\n id: timer-log\n from:\n uri: timer\n parameters:\n timerName: tick\n period: \"{{timer.period}}\"\n steps:\n - setBody:\n expression:\n simple:\n expression: \"${random(0,40)}\"\n - choice:\n when:\n - simple: \"${body} >= 30\"\n steps:\n - to:\n uri: seda:hot\n otherwise:\n [...] + "application.properties": "timer.period=5000\ngreeting.message=Hello Camel!\n" + } + }, + { + "id": 5, + "request": "Add a second route with id hot-consumer that consumes from seda:hot and logs \"hot from <the route id>: <the body>\".", + "check": { + "file_regex": "hot-consumer", + "log_regex": "^(hot from hot-consumer: \\d+|normal: \\d+)", + "min_log": 1, + "routes": 2 + }, + "reference": { + "timer-log.camel.yaml": "- route:\n id: timer-log\n from:\n uri: timer\n parameters:\n timerName: tick\n period: \"{{timer.period}}\"\n steps:\n - setBody:\n expression:\n simple:\n expression: \"${random(0,40)}\"\n - choice:\n when:\n - simple: \"${body} >= 30\"\n steps:\n - to:\n uri: seda:hot\n otherwise:\n [...] + "application.properties": "timer.period=5000\ngreeting.message=Hello Camel!\n" + } + }, + { + "id": 6, + "request": "Add error handling so that if any exception happens in a route, the error is logged and the message continues without failing.", + "check": { + "file_regex": "onException|errorHandler", + "log_regex": "^(hot from hot-consumer: \\d+|normal: \\d+)", + "min_log": 1, + "routes": 2 + }, + "reference": { + "timer-log.camel.yaml": "- onException:\n exception:\n - java.lang.Exception\n continued:\n constant: \"true\"\n steps:\n - log:\n message: \"error: ${exception.message}\"\n- route:\n id: timer-log\n from:\n uri: timer\n parameters:\n timerName: tick\n period: \"{{timer.period}}\"\n steps:\n - setBody:\n expression:\n simple:\n expression: \"${random(0,40)}\"\n - c [...] + "application.properties": "timer.period=5000\ngreeting.message=Hello Camel!\n" + } + }, + { + "id": 7, + "request": "Move the 30 threshold into application.properties as hot.threshold and use it from the route.", + "check": { + "file_regex": "\\{\\{hot\\.threshold\\}\\}|\\$\\{properties:hot\\.threshold\\}", + "props_regex": "hot\\.threshold\\s*=\\s*30", + "log_regex": "^(hot from hot-consumer: \\d+|normal: \\d+)", + "min_log": 1, + "routes": 2 + }, + "reference": { + "timer-log.camel.yaml": "- onException:\n exception:\n - java.lang.Exception\n continued:\n constant: \"true\"\n steps:\n - log:\n message: \"error: ${exception.message}\"\n- route:\n id: timer-log\n from:\n uri: timer\n parameters:\n timerName: tick\n period: \"{{timer.period}}\"\n steps:\n - setBody:\n expression:\n simple:\n expression: \"${random(0,40)}\"\n - c [...] + "application.properties": "timer.period=5000\ngreeting.message=Hello Camel!\nhot.threshold=30\n" + } + }, + { + "id": 8, + "request": "Rename the first route id from timer-log to sensor, and make its normal log step use the logger name sensor.", + "check": { + "file_regex": "id: sensor", + "file_regex2": "logName: sensor|loggerName: sensor|logger: sensor", + "log_regex": "^(hot from hot-consumer: \\d+|normal: \\d+)", + "min_log": 1, + "routes": 2 + }, + "reference": {} + } + ] +} diff --git a/ai-benchmark/stepwise-project/application.properties b/ai-benchmark/stepwise-project/application.properties new file mode 100644 index 0000000..370577a --- /dev/null +++ b/ai-benchmark/stepwise-project/application.properties @@ -0,0 +1,4 @@ +# Timer period in milliseconds +timer.period=1000 +# Greeting message to log +greeting.message=Hello Camel! diff --git a/ai-benchmark/stepwise-project/timer-log.camel.yaml b/ai-benchmark/stepwise-project/timer-log.camel.yaml new file mode 100644 index 0000000..3e95c9e --- /dev/null +++ b/ai-benchmark/stepwise-project/timer-log.camel.yaml @@ -0,0 +1,15 @@ +- route: + id: timer-log + from: + uri: timer + parameters: + timerName: tick + period: "{{timer.period}}" + steps: + - setBody: + expression: + simple: + expression: "{{greeting.message}}" + - log: + message: "${body}" + diff --git a/ai-benchmark/summarize_runs.py b/ai-benchmark/summarize_runs.py new file mode 100755 index 0000000..93001a2 --- /dev/null +++ b/ai-benchmark/summarize_runs.py @@ -0,0 +1,78 @@ +#!/usr/bin/env python3 +"""Markdown summary of completed runs: one-shot and stepwise results plus the suite wall clock. + +Usage: summarize_runs.py <tag> [<tag> ...] + +A run with tag T has its one-shot results in oneshot-T/<example>/result.json (BENCH_OUT=oneshot-T), its stepwise +results in stepwise/T/results.json (BENCH_TAG=T) and its wall clock in T.log (written by run-suite.sh). +""" +import json, os, re, sys +HERE = os.path.dirname(os.path.abspath(__file__)) +names = [e["name"] for e in json.load(open(os.path.join(HERE, "examples.json")))] +tags = sys.argv[1:] +if not tags: + print(__doc__); sys.exit(1) +def wall(tag): + p = os.path.join(HERE, tag + ".log") + if not os.path.exists(p): + return None + t = {} + for l in open(p): + m = re.search(rf"\[{re.escape(tag)}\] (one-shot|stepwise) (start|done) (\d+):(\d+):(\d+)", l) + if m: + t[(m.group(1), m.group(2))] = int(m.group(3)) * 3600 + int(m.group(4)) * 60 + int(m.group(5)) + try: + def span(x): + d = t[(x, "done")] - t[(x, "start")] + return d + 86400 if d < 0 else d # a run that crosses midnight + return span("one-shot"), span("stepwise") + except KeyError: + return None +STALL = 600 # a single model call longer than this is the machine asleep, not the model +def stalls(paths): + total = 0.0 + for p in paths: + if not os.path.exists(p): + continue + for l in open(p): + try: + o = json.loads(l) + except json.JSONDecodeError: + continue + if isinstance(o.get("secs"), (int, float)) and o["secs"] > STALL: + total += o["secs"] + return total +rows = {} +for tag in tags: + d = os.path.join(HERE, "oneshot-" + tag) + one = [json.load(open(os.path.join(d, n, "result.json"))) for n in names if os.path.exists(os.path.join(d, n, "result.json"))] + sp = os.path.join(HERE, "stepwise", tag, "results.json") + st = json.load(open(sp)) if os.path.exists(sp) else [] + w = wall(tag) + st1 = stalls(os.path.join(d, n, "trace.jsonl") for n in names) + st2 = stalls(os.path.join(HERE, "stepwise", tag, f"step{i}.trace.jsonl") for i in range(1, len(st) + 1)) + if w and (st1 or st2): + w = (int(w[0] - st1), int(w[1] - st2)) + rows[tag] = dict( + n=len(one), stall=int(st1 + st2), + ok=sum(x["ok"] for x in one), first=sum(1 for x in one if x["ok"] and x["rounds"] == 1), + rounds=sum(x["rounds"] for x in one), calls=sum(x["tool_calls"] for x in one), + tok=sum(x["tokens"] for x in one), + sn=len(st), sok=sum(x["ok"] for x in st), + slen=sum(1 for x in st if x["ok"] or (x["file_ok"] and x["props_ok"] and x["errors"] == 0)), + scalls=sum(x["tool_calls"] for x in st), srefused=sum(x["refused_writes"] for x in st), stok=sum(x["tokens"] for x in st), + wall=(f"{(w[0]+w[1])//60} min ({w[0]//60} + {w[1]//60})" if w else "?"), + fails=[n for n, x in zip(names, one) if not x["ok"]], sfails=[i + 1 for i, x in enumerate(st) if not x["ok"]]) +def line(label, f): + return "| " + label + " | " + " | ".join(f(rows[t]) for t in tags) + " |" +print("| | " + " | ".join(tags) + " |"); print("|---|" + "---|" * len(tags)) +print(line("One-shot passes", lambda x: f"{x['ok']} of {x['n']}")) +print(line("One-shot first-round passes", lambda x: str(x["first"]))) +print(line("One-shot rounds / tool calls", lambda x: f"{x['rounds']} / {x['calls']}")) +print(line("One-shot tokens", lambda x: f"{x['tok']:,}")) +print(line("Stepwise passes (strict / lenient)", lambda x: f"{x['sok']} / {x['slen']} of {x['sn']}")) +print(line("Stepwise tool calls / refused writes", lambda x: f"{x['scalls']} / {x['srefused']}")) +print(line("Stepwise tokens", lambda x: f"{x['stok']:,}")) +print(line("Suite wall clock (sleep stalls removed)", lambda x: x["wall"] + (f", {x['stall']//60} min asleep" if x["stall"] else ""))) +print(line("One-shot failures", lambda x: ", ".join(x["fails"]) or "none")) +print(line("Stepwise failures", lambda x: ", ".join(map(str, x["sfails"])) or "none"))
