This is an automated email from the ASF dual-hosted git repository.
davsclaus pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/camel.git
The following commit(s) were added to refs/heads/main by this push:
new 42060dc5ce74 CAMEL-26875: camel-jbang - AI tools: the answer to a
repeated read says what to do next (#29697)
42060dc5ce74 is described below
commit 42060dc5ce74fd50680c27707dab0351f5f5e75c
Author: Claus Ibsen <[email protected]>
AuthorDate: Thu Nov 12 13:09:12 2026 +0100
CAMEL-26875: camel-jbang - AI tools: the answer to a repeated read says
what to do next (#29697)
The note for the third and later identical call of a deterministic tool now
starts with the next step, then says the answer is the same each time; its
count is timesAsked. camel_catalog_sample and camel_catalog_find get a hint
(write the file, which checks it), and camel_catalog_doc says to go on with
the options it gave. In the local-model benchmark (r32), a model got the old
note, which only said the answer was already given, and asked again until
its tool calls ran out, without writing the test.
Claude-Session: https://claude.ai/code/session_016Y4dDi5iQNgtHrDdz2NpwF
Co-authored-by: Claude Opus 5.5 <[email protected]>
---
.../modules/ROOT/pages/camel-jbang-mcp.adoc | 12 ++--
.../dsl/jbang/core/commands/ai/AuthoringTools.java | 22 ++++--
.../jbang/core/commands/ai/RepeatedToolCalls.java | 18 +++--
.../dsl/jbang/core/commands/ai/ToolDescriptor.java | 6 +-
.../core/commands/ai/RepeatedToolCallsTest.java | 78 ++++++++++++++++++++--
.../core/commands/mcp/AuthoringToolsTest.java | 11 ++-
6 files changed, 122 insertions(+), 25 deletions(-)
diff --git a/docs/user-manual/modules/ROOT/pages/camel-jbang-mcp.adoc
b/docs/user-manual/modules/ROOT/pages/camel-jbang-mcp.adoc
index 67415f5f6890..2fb95d1b16f0 100644
--- a/docs/user-manual/modules/ROOT/pages/camel-jbang-mcp.adoc
+++ b/docs/user-manual/modules/ROOT/pages/camel-jbang-mcp.adoc
@@ -541,11 +541,15 @@ with `"_meta": {"camel.apache.org/deterministic": true}`,
so a client can notice
An agent asking one of them the same question again and again gets nothing
new, and a small local model can get
stuck doing that until its tool call budget is spent. So the first two
identical calls of a session are answered
-in full, and from the third the answer is a short note instead: that the call
was already answered and how many
-times it was asked. The changed answer helps the agent break the loop, and the
context is not filled with copies.
+in full, and from the third the answer is a short note instead. The note
starts with what to do next. Then it says
+how many times the call was asked, and that the answer is the same each time
and is in the earlier tool result.
+The changed answer helps the agent break the loop, and the context is not
filled with copies.
+The next step depends on the tool. For `camel_catalog_sample` it is to write
the file with `camel_write_file`,
+which checks the file and says what to change. For `camel_catalog_find` it is
to use a match, or to ask with
+another term, and then write the file. For `camel_catalog_doc` it is to go on
with the options it gave; when a page
+was cut, `docPage=<section title>` returns a section it left out.
`camel_validate_source` counts the same way when it is given `content` and no
`directory`: the same source gets
-the same answer, and the note also says to change the line the error names.
For `camel_catalog_doc` the note says
-how to get the sections that a cut page left out: `docPage=<section title>`
returns one section. Validating a file on disk, or
+the same answer, and the note says to change the line the error names.
Validating a file on disk, or
`content` with a `directory` (the checks then read the other files of the
project, which a fix can change), is
always answered in full.
diff --git
a/dsl/camel-jbang/camel-jbang-core/src/main/java/org/apache/camel/dsl/jbang/core/commands/ai/AuthoringTools.java
b/dsl/camel-jbang/camel-jbang-core/src/main/java/org/apache/camel/dsl/jbang/core/commands/ai/AuthoringTools.java
index f7355e56b2c3..57d9e74d1607 100644
---
a/dsl/camel-jbang/camel-jbang-core/src/main/java/org/apache/camel/dsl/jbang/core/commands/ai/AuthoringTools.java
+++
b/dsl/camel-jbang/camel-jbang-core/src/main/java/org/apache/camel/dsl/jbang/core/commands/ai/AuthoringTools.java
@@ -131,8 +131,11 @@ public final class AuthoringTools {
.param("camelVersion", "string", VERSION_DESC, false)
.core(true)
.deterministic(true)
- .repeatHint("When that answer was cut, it named the sections
it left out: docPage=<section title> returns"
- + " one of them.")
+ // r32: three doc calls were each made 10 to 21 times with the
same arguments; the write checks the
+ // options, so writing is the next step (CAMEL-26875)
+ .repeatHint("Go on with the options the earlier answer gave:
write the file with camel_write_file, which"
+ + " checks the options against the catalog. When
that answer was cut, it named the sections"
+ + " it left out: docPage=<section title> returns
one of them.")
.executor((ctx, args) -> {
applyVersion(ctx, args);
return CatalogDocs.catalogDoc(ctx.catalog(),
args.get("name"), args.get("endpoint"),
@@ -154,6 +157,8 @@ public final class AuthoringTools {
.param("camelVersion", "string", VERSION_DESC, false)
.core(true)
.deterministic(true)
+ .repeatHint("Use a match from the earlier answer, or ask with
another term; then write the file with"
+ + " camel_write_file.")
.executor((ctx, args) -> {
applyVersion(ctx, args);
return CatalogDocs.find(ctx.catalog(), args.get("term"),
args.get("kind"),
@@ -176,6 +181,12 @@ public final class AuthoringTools {
.param("camelVersion", "string", VERSION_DESC, false)
.core(true)
.deterministic(true)
+ // r32: a sample that does not exist asked for again and again
in a Citrus test step, until the calls of
+ // the step were spent and no test was written; the write
checks the test and names the actions Citrus
+ // has (CAMEL-26875)
+ .repeatHint("Write the file now with camel_write_file: when
the earlier answer had no sample, asking again"
+ + " gives none either. The write checks the file,
and its answer says what to change (for a"
+ + " Citrus test: which actions it takes).")
.executor((ctx, args) -> {
applyVersion(ctx, args);
return CatalogSamples.sample(ctx.catalog(),
args.get("kind"), args.get("name"),
@@ -195,8 +206,8 @@ public final class AuthoringTools {
// the same content gets the same answer: a model that
validates it again and again is stuck
// (CAMEL-25371). Not with a directory: the checks then read
the other files, which a fix can change.
.deterministicWhen("content", "directory")
- .repeatHint("The content is the same each time, so the answer
is too: change the line the error names,"
- + " then validate the changed content, or write it
with camel_write_file.")
+ .repeatHint("Change the line the error names, then validate
the changed content, or write it with"
+ + " camel_write_file.")
.executor((ctx, args) -> {
applyVersion(ctx, args);
String file = required(args, "file");
@@ -414,8 +425,7 @@ public final class AuthoringTools {
// the same expression on the same message gets the same
answer, unless it reads the clock, a random
// value or the integration: a model that evaluates it again
and again is stuck
.deterministicIf(ExpressionEvaluator::sameAnswer)
- .repeatHint("The same expression on the same message gives the
same result: change the expression or"
- + " the message (body, headers, variables) to
learn something new.")
+ .repeatHint("Change the expression or the message (body,
headers, variables) to learn something new.")
.executor((ctx, args) -> {
String expression = required(args, "expression");
Map<String, Object> headers =
ExpressionEvaluator.parseValues("headers", args.get("headers"));
diff --git
a/dsl/camel-jbang/camel-jbang-core/src/main/java/org/apache/camel/dsl/jbang/core/commands/ai/RepeatedToolCalls.java
b/dsl/camel-jbang/camel-jbang-core/src/main/java/org/apache/camel/dsl/jbang/core/commands/ai/RepeatedToolCalls.java
index 2b1350d075f2..ca7f56b57ef7 100644
---
a/dsl/camel-jbang/camel-jbang-core/src/main/java/org/apache/camel/dsl/jbang/core/commands/ai/RepeatedToolCalls.java
+++
b/dsl/camel-jbang/camel-jbang-core/src/main/java/org/apache/camel/dsl/jbang/core/commands/ai/RepeatedToolCalls.java
@@ -33,6 +33,13 @@ import org.apache.camel.util.json.JsonObject;
* the input does change and the context is not filled with copies. The first
two calls are always answered in full, so
* a legitimate re-read still works.
* <p>
+ * The note starts with what to do next, and then says that the answer is the
same each time. A note that only said the
+ * answer was already given did not stop a small model. In the local-model
benchmark (r32), a model asked for a sample
+ * that does not exist, and then asked the same 11 or 12 more times, in 3 runs
of a step. It said sorry for the repeated
+ * calls and asked again, until its 25 tool calls were spent. It never wrote
the test of the step. So the note starts
+ * with the next step: the {@link ToolDescriptor#repeatHint() hint} of the
tool, such as writing the file, whose checks
+ * say what to change, or else to go on with the task.
+ * <p>
* One instance tracks one session: a client connection, or a conversation of
an AI panel. Call {@link #reset()} when a
* new session starts on the same instance.
*/
@@ -44,6 +51,9 @@ public final class RepeatedToolCalls {
/** How many identical calls are answered in full before the short note is
given. */
static final int FULL_ANSWERS = 2;
+ /** The next step the note starts with when the tool has no {@link
ToolDescriptor#repeatHint() hint} of its own. */
+ static final String NEXT_STEP = "Go on with the next step of your task.";
+
/** Guards against unbounded growth over a very long session; the oldest
counts are simply forgotten. */
private static final int MAX_TRACKED = 1000;
@@ -76,10 +86,10 @@ public final class RepeatedToolCalls {
note.put("repeated", true);
note.put("tool", tool.name());
note.put("timesAsked", times);
- note.put("note", tool.name() + " was already called with these same
arguments " + (times - 1)
- + " times in this session. Its answer does not
change, so it is not sent again: it is in your"
- + " earlier tool result."
- + (tool.repeatHint() != null ? " " +
tool.repeatHint() : ""));
+ // the next step first, then the fact; the count is timesAsked, this
call included (CAMEL-26875)
+ note.put("note", (tool.repeatHint() != null ? tool.repeatHint() :
NEXT_STEP) + " You asked this " + times
+ + " times now with the same arguments, and the answer
is the same each time: it is in your"
+ + " earlier tool result.");
return note;
}
diff --git
a/dsl/camel-jbang/camel-jbang-core/src/main/java/org/apache/camel/dsl/jbang/core/commands/ai/ToolDescriptor.java
b/dsl/camel-jbang/camel-jbang-core/src/main/java/org/apache/camel/dsl/jbang/core/commands/ai/ToolDescriptor.java
index 3a9921263ca0..dfaee5fe0e11 100644
---
a/dsl/camel-jbang/camel-jbang-core/src/main/java/org/apache/camel/dsl/jbang/core/commands/ai/ToolDescriptor.java
+++
b/dsl/camel-jbang/camel-jbang-core/src/main/java/org/apache/camel/dsl/jbang/core/commands/ai/ToolDescriptor.java
@@ -148,7 +148,11 @@ public class ToolDescriptor {
return this;
}
- /** What the note of a repeated call adds for this tool: what to do
instead of asking again. */
+ /**
+ * What the note of a repeated call starts with for this tool: the next
step instead of asking again, such as
+ * writing the file, written as one or more whole sentences. Without a
hint the note says to go on with the next
+ * step of the task, see {@link RepeatedToolCalls}.
+ */
public ToolDescriptor repeatHint(String hint) {
repeatHint = hint;
return this;
diff --git
a/dsl/camel-jbang/camel-jbang-core/src/test/java/org/apache/camel/dsl/jbang/core/commands/ai/RepeatedToolCallsTest.java
b/dsl/camel-jbang/camel-jbang-core/src/test/java/org/apache/camel/dsl/jbang/core/commands/ai/RepeatedToolCallsTest.java
index 6fb0029a7e71..c7bd71cb5a79 100644
---
a/dsl/camel-jbang/camel-jbang-core/src/test/java/org/apache/camel/dsl/jbang/core/commands/ai/RepeatedToolCallsTest.java
+++
b/dsl/camel-jbang/camel-jbang-core/src/test/java/org/apache/camel/dsl/jbang/core/commands/ai/RepeatedToolCallsTest.java
@@ -26,6 +26,10 @@ import static org.assertj.core.api.Assertions.assertThat;
class RepeatedToolCallsTest {
+ /** What the note says after the next step, on the third identical call. */
+ private static final String ASKED_3_TIMES = "You asked this 3 times now
with the same arguments, and the answer is"
+ + " the same each time: it is
in your earlier tool result.";
+
private final RepeatedToolCalls calls = new RepeatedToolCalls();
private final ToolDescriptor catalogDoc =
ToolRegistry.findTool("camel_catalog_doc");
@@ -47,11 +51,73 @@ class RepeatedToolCallsTest {
assertThat(note.getBoolean("repeated")).isTrue();
assertThat(note.getString("tool")).isEqualTo("camel_catalog_doc");
assertThat(note.getInteger("timesAsked")).isEqualTo(3);
- assertThat(note.getString("note")).contains("2 times");
+ // the next step first, then the fact, with the count of timesAsked
(CAMEL-26875)
+ assertThat(note.getString("note"))
+ .startsWith("Go on with the options the earlier answer gave:
write the file with camel_write_file")
+ .endsWith(ASKED_3_TIMES);
// a page that was cut was not answered in full: the note says how to
get the sections it left out
assertThat(note.getString("note")).doesNotContain("in
full").contains("docPage=<section title>");
- assertThat(calls.repeatOf(catalogDoc,
sql).getInteger("timesAsked")).isEqualTo(4);
+ JsonObject fourth = calls.repeatOf(catalogDoc, sql);
+ assertThat(fourth.getInteger("timesAsked")).isEqualTo(4);
+ assertThat(fourth.getString("note")).contains("You asked this 4 times
now");
+ }
+
+ /**
+ * r32, a Citrus test step: camel_catalog_sample had no sample of 'send
message' as an EIP, and the model asked the
+ * same 11 or 12 more times, until its tool calls were spent, without
writing the test. The note starts with the
+ * next step: write the file, whose checks say what to change
(CAMEL-26875).
+ */
+ @Test
+ void aRepeatedSampleCallIsToldToWriteTheFile() {
+ ToolDescriptor sample = ToolRegistry.findTool("camel_catalog_sample");
+ Map<String, String> sendMessage = Map.of("name", "send message",
"kind", "eip");
+ successfulCall(sample, sendMessage);
+ successfulCall(sample, sendMessage);
+
+ JsonObject note = calls.repeatOf(sample, sendMessage);
+ assertThat(note.getString("tool")).isEqualTo("camel_catalog_sample");
+ assertThat(note.getInteger("timesAsked")).isEqualTo(3);
+ assertThat(note.getString("note")).isEqualTo("Write the file now with
camel_write_file: when the earlier answer"
+ + " had no sample, asking
again gives none either. The write checks"
+ + " the file, and its
answer says what to change (for a Citrus test:"
+ + " which actions it
takes). " + ASKED_3_TIMES);
+ }
+
+ @Test
+ void aRepeatedFindIsToldToUseAMatchOrAnotherTerm() {
+ ToolDescriptor find = ToolRegistry.findTool("camel_catalog_find");
+ Map<String, String> term = Map.of("term", "send message");
+ successfulCall(find, term);
+ successfulCall(find, term);
+
+ assertThat(calls.repeatOf(find, term).getString("note"))
+ .isEqualTo("Use a match from the earlier answer, or ask with
another term; then write the file with"
+ + " camel_write_file. " + ASKED_3_TIMES);
+ }
+
+ @Test
+ void aToolWithoutAHintIsToldToGoOnWithTheTask() {
+ ToolDescriptor diagnose =
ToolRegistry.findTool("camel_error_diagnose");
+ assertThat(diagnose.repeatHint()).isNull();
+ Map<String, String> error = Map.of("error", "No endpoint could be
found for: xyz:test");
+ successfulCall(diagnose, error);
+ successfulCall(diagnose, error);
+
+ assertThat(calls.repeatOf(diagnose, error).getString("note"))
+ .isEqualTo("Go on with the next step of your task. " +
ASKED_3_TIMES);
+ }
+
+ /** The note starts with the hint of the tool, and a sentence of its own
follows: a hint is whole sentences. */
+ @Test
+ void everyHintIsWholeSentences() {
+ for (ToolDescriptor tool : ToolRegistry.allTools()) {
+ String hint = tool.repeatHint();
+ if (hint != null) {
+
assertThat(Character.isUpperCase(hint.charAt(0))).as(tool.name() + ": " +
hint).isTrue();
+ assertThat(hint).as(tool.name()).endsWith(".");
+ }
+ }
}
/**
@@ -68,10 +134,8 @@ class RepeatedToolCallsTest {
successfulCall(eval, args);
JsonObject note = calls.repeatOf(eval, args);
assertThat(note).isNotNull();
- assertThat(note.getString("note")).startsWith("camel_eval_expression
was already called with these same arguments 2"
- + " times")
- .endsWith("The same expression on the same message gives the
same result: change the expression or the"
- + " message (body, headers, variables) to learn
something new.");
+ assertThat(note.getString("note")).isEqualTo("Change the expression or
the message (body, headers, variables) to"
+ + " learn something new.
" + ASKED_3_TIMES);
// another message is another question
assertThat(calls.repeatOf(eval, Map.of("expression",
"body.getClass().getName()", "language", "groovy", "body",
"[]"))).isNull();
@@ -152,7 +216,7 @@ class RepeatedToolCallsTest {
successfulCall(validate, args);
JsonObject note = calls.repeatOf(validate, args);
assertThat(note).isNotNull();
- assertThat(note.getString("note")).contains("change the line the error
names");
+ assertThat(note.getString("note")).startsWith("Change the line the
error names");
Map<String, String> fromDisk = Map.of("file", "route.camel.yaml");
for (int i = 0; i < 4; i++) {
diff --git
a/dsl/camel-jbang/camel-jbang-mcp/src/test/java/org/apache/camel/dsl/jbang/core/commands/mcp/AuthoringToolsTest.java
b/dsl/camel-jbang/camel-jbang-mcp/src/test/java/org/apache/camel/dsl/jbang/core/commands/mcp/AuthoringToolsTest.java
index 7873744e3103..c7e1b4f6a3b2 100644
---
a/dsl/camel-jbang/camel-jbang-mcp/src/test/java/org/apache/camel/dsl/jbang/core/commands/mcp/AuthoringToolsTest.java
+++
b/dsl/camel-jbang/camel-jbang-mcp/src/test/java/org/apache/camel/dsl/jbang/core/commands/mcp/AuthoringToolsTest.java
@@ -277,7 +277,10 @@ class AuthoringToolsTest {
assertThat(second).isEqualTo(first);
assertThat(third.getBoolean("repeated")).isTrue();
assertThat(third.getInteger("timesAsked")).isEqualTo(3);
- assertThat(third.getString("note")).contains("camel_catalog_doc", "2
times");
+ assertThat(third.getString("tool")).isEqualTo("camel_catalog_doc");
+ // the next step first, then the fact (CAMEL-26875)
+ assertThat(third.getString("note")).startsWith("Go on with the options
the earlier answer gave")
+ .contains("You asked this 3 times now");
assertThat(third.toJson().length()).isLessThan(first.toJson().length()
/ 4);
// another question, or another connection, is answered in full
@@ -302,7 +305,8 @@ class AuthoringToolsTest {
assertThat(first.get("repeated")).isNull();
assertThat(second.get("repeated")).isNull();
assertThat(third.getBoolean("repeated")).isTrue();
- assertThat(third.getString("note")).contains("camel_validate_source",
"change the line the error names");
+ assertThat(third.getString("tool")).isEqualTo("camel_validate_source");
+ assertThat(third.getString("note")).startsWith("Change the line the
error names");
// changed content is a new question
assertThat(counted.camel_validate_source(agent, null, null,
"route.camel.yaml", content + "\n", null)
.get("repeated")).isNull();
@@ -325,7 +329,8 @@ class AuthoringToolsTest {
JsonObject third = counted.camel_eval_expression(agent,
"${body.length()}", null, "[1, 2]", null, null, null,
null, null);
assertThat(third.getBoolean("repeated")).isTrue();
-
assertThat(third.getString("note")).contains("camel_eval_expression", "change
the expression or the message");
+
assertThat(third.getString("tool")).isEqualTo("camel_eval_expression");
+ assertThat(third.getString("note")).startsWith("Change the
expression or the message");
// a random value is asked again for a new one
for (int i = 0; i < 3; i++) {
assertThat(counted.camel_eval_expression(agent,
"${random(1,10)}", null, null, null, null, null, null, null)
