davsclaus opened a new pull request, #29904:
URL: https://github.com/apache/camel/pull/29904

   ## What
   
   A follow-up to #29890 (design: Camel CLI short commands without a JVM 
start). It adds the result of the benchmark that the design asked for. Docs 
only.
   
   The local-model benchmark (qwen3.6:35b, 53 beginner questions about 5 
example apps, 3 rounds, 159 answers per run) ran without and with an 
instruction at the start of each conversation: look options up with 
`camel_catalog_doc`, and check every snippet with `camel_validate_source` 
before you answer.
   
   | per run of 159 answers | without | with |
   |---|---|---|
   | pass | 130 | 128 |
   | "what does this app do" (always the first question) | 8/15 | 12/15 |
   | "how can I make it better" | 12/15 | 5/15 |
   | code blocks in the answers | 84 | 221 |
   | snippets validated | 0 | 0 |
   | options that do not exist | 6 in 4 answers | 32 in 14 answers |
   | compactions of the history | 6 | 13 |
   | model time per answer | 5.8 s | 8.1 s |
   
   The instruction faded after the first question. Most of the gain on the 
first question is one app: in the first run the model answered from the 
AI-written summary that the readme tool returned then (fixed by CAMEL-26946, 
#29891).
   
   ## Change
   
   - **Motivation:** the two runs, with the caveat that this is one model and 
one run each.
   - **Summary and recommendation:** telling an agent to check is not enough, 
so the checks run without its choice.
     - The AI panel and `camel ask` mark the mistakes in the code of an answer 
(CAMEL-26960).
     - The MCP server already validates a file before `camel_write_file` writes 
it.
     - An agent that edits files with its own tools needs a hook that runs 
`camel validate yaml` on each edited route file. That hook runs on every edit, 
so it is where the daemon saves the most.
   - **Measure the agents:** the next runs are a line in the system prompt of 
the AI panel (two jars that differ only in that line) and a run with the marks 
of CAMEL-26960.
   - Links CAMEL-26961 (the daemon itself) and CAMEL-26960.
   
   _Claude Code on behalf of Claus Ibsen (davsclaus)_
   
   🤖 Generated with [Claude Code](https://claude.com/claude-code)
   
   https://claude.ai/code/session_012nJsFsjqqFiwroeHRWaKbS
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to