gnodet-bot commented on code in PR #29904: URL: https://github.com/apache/camel/pull/29904#discussion_r4572490600
########## design/cli-short-commands.adoc: ########## @@ -77,6 +82,31 @@ got it wrong; with a hint in the question it read the doc and got it right. It s exist, and validated nothing in 9 answers. A lookup also costs context: the doc of the file component is 17K characters, and three docs in one answer forced a compaction of the history. +Two full runs on the night of 2026-11-13 then asked the same model 53 questions about 5 example apps, 3 times each, +the second with an instruction at the start of each conversation: look options up with `camel_catalog_doc`, and check +every snippet with `camel_validate_source` before you answer. Per run of 159 answers, without and with the +instruction: Review Comment: ⚠️ **Sentence too long (~47 words):** The opening sentence of the benchmark section runs all the way through line 88. Split at "3 times each" to separate the setup from the method: ```suggestion Two full runs on the night of 2026-11-13 asked the same model 53 questions about 5 example apps, 3 times each. The second run added an instruction at the start of each conversation: look options up with `camel_catalog_doc`, and check every snippet with `camel_validate_source` before you answer. Per run of 159 answers, without and with the instruction: ``` ########## design/cli-short-commands.adoc: ########## @@ -38,8 +40,11 @@ against 10-40 ms estimated), for a fraction of the build and release work. The seconds a daemon saves are small next to the turns of the model. The reason to build it is that a command that answers at once can be called after every step. An agent makes fewer mistakes when it validates after every edit, looks an option up with `camel doc` instead of guessing it, and checks `camel status` after every reload. Speed alone -does not make an agent do that, it has to be told (see the motivation), but speed makes it cheap to do. With the Java -client the daemon answers in 50-65 ms, under the 0.1 s at which a response feels instant, like a native command. +does not make an agent do that, and telling it to is not enough: in a benchmark with a local model, an instruction to +check made the model read more, but it checked none of the code it wrote (see the motivation). The checks have to run Review Comment: ⚠️ **Sentence too long (~42 words):** This sentence joins two independent observations with a colon. Split at the colon to give each idea its own sentence: ```suggestion does not make an agent do that, and telling it to is not enough. In a benchmark with a local model, an instruction to check made the model read more, but it checked none of the code it wrote (see the motivation). The checks have to run whether the model asks for them or not, for example in a hook of the agent that validates every edited file, and a ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
