davsclaus opened a new pull request, #29906:
URL: https://github.com/apache/camel/pull/29906
## What
CAMEL-26960, step 1. The code in an AI answer is what a user copies, but
only its simple expressions were checked (CAMEL-24805). A benchmark with a
local model (qwen3.6:35b, 159 answers per run) told the model to validate every
snippet: it wrote 2.6 times as many code blocks, validated none of them, and
options that do not exist went from 6 to 32. So the AI panel and `camel ask`
now check the code blocks of every answer against the catalog, and mark what
does not exist.
## Change
**What is checked** (`AnswerChecks.checkCode`, camel-jbang-core):
- YAML: unknown keys and options, from the YAML DSL schema.
- Endpoint uris: options the component does not have.
- `camel.*` keys of properties lines, in a block of any language.
- A value of the wrong type, only when the validator has a hint for it, such
as `onException` written as a list with the exception class as a key.
**Fragments** (`YamlFragment`): most code in an answer is a fragment, which
the schema refuses for its shape before it looks at the keys. So each one is
wrapped in what it belongs to:
- a `from:` without its route goes in a route file;
- a list of steps, or a single EIP, goes in the steps of a route;
- `redeliveryPolicy:` goes under an `onException`, and
`resilience4jConfiguration:` and `onFallback:` under a `circuitBreaker`.
**What is not marked:** code in an answer leaves things out on purpose, and
a wrong mark costs trust in the others. So none of these are marked:
- a required option left out, a placeholder, a bare "string found, object
expected";
- an unknown key where the wrapping put the fragment, such as options shown
without their EIP (`slidingWindowSize: 10`);
- parse errors, whose line and column point into the wrapping.
**Where the marks show:**
- **AI panel:** a line under the block, `` ✗ line 4: `multiplier` is not an
option of `redeliveryPolicy` ``, with the hint of the validator when it has
one. *Copy* still copies only the code.
- **`camel ask`:** a `Check: code block 1, line 4: ...` line on stderr, so
stdout is still only the answer.
- **No correction turn**, as decided on the JIRA. The model is told of the
marks with the next question, and of a simple expression it did not fix, which
before was only shown. Its own answer stays in the history as it wrote it.
**Two validator bugs**, found when the check ran over the benchmark answers.
Both also affected `camel_validate_source`:
- A commented-out option under `parameters:` was reported as the unknown
option `'# idempotentRepository'` (`EndpointChecks`).
- A bean with its `name:` and an unknown key was told "the name is a
property, not the key" (`SchemaHints`). A `parameters:` key now gets "goes
under properties: or constructors:".
**Size:** `AiPanel.java` is at 3197 lines, under its `SourceSizeTest` cap of
3200.
**Docs:** the AI panel page (`camel-jbang-tui-ai.adoc`) and the `camel ask`
page (`camel-jbang-ai.adoc`).
**Not in this PR**, the next steps on the JIRA:
- option maps without their key;
- unknown `camel.*` groups (`camel.tracing.enabled`, `camel.profile`); the
catalog does not know `camel.servlet.*` of Spring Boot either, so a plain check
would mark it;
- the answers of ACP agents;
- a markdown mode of `camel_validate_source` for MCP clients.
## Measured
The check ran over the 318 answers of two benchmark runs (305 code blocks).
It made 53 marks in 27 answers, and I read each one: all are real mistakes.
Examples:
- `messageIdRepository` on `idempotentConsumer`;
- `forceOpen`, `openDuration` and `failureThreshold` on `circuitBreaker`;
- `multiplier` and `exponentialBackoffMultiplier` on `redeliveryPolicy`;
- `deleteAfterRead` on `file`, `retryCount` on `direct`;
- `steps` under `route`;
- `camel.server.context-path`.
The whole run took 0.75 s, validator start included.
## Tests
- `AnswerCodeChecksTest` (new, 9 tests), with snippets from the benchmark:
- marks that must come: from, steps, an EIP, `redeliveryPolicy`, endpoint
options, properties, `onException` as a list;
- blocks that must not be marked: a valid route, an option map without its
key, placeholders, docker-compose, Kubernetes, Spring Boot settings, a message
body, an unterminated block;
- where the marks go, and what the model is told.
- `AiPanelTest`: the mark is under the block, the model is not asked again,
and the next question carries the note.
- `AskLoopTest`: the mark is on stderr.
- `SourceValidatorEndpointTest` and `YamlValidatorPropertyHintTest`: the two
bug fixes. Both tests fail on the code before the change.
- Full suites pass: camel-yaml-dsl-validator 370, camel-jbang-core 2999 +
254, camel-jbang-plugin-tui 2575.
_Claude Code on behalf of Claus Ibsen (davsclaus)_
🤖 Generated with [Claude Code](https://claude.com/claude-code)
https://claude.ai/code/session_012nJsFsjqqFiwroeHRWaKbS
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]