Claus Ibsen created CAMEL-26667:
-----------------------------------
Summary: camel-chaos - findings from the hands-on round of stage
1: report accuracy, feedback on rules, live rule state, new faults
Key: CAMEL-26667
URL: https://issues.apache.org/jira/browse/CAMEL-26667
Project: Camel
Issue Type: Improvement
Components: camel-jbang
Reporter: Claus Ibsen
Fix For: 4.24.0
h2. Hands-on round 1 of camel-chaos (stage 1): findings and ideas
Stage 1 of camel-chaos (CAMEL-26330) was tried for real with {{camel run
--chaos}}, driven from the Camel CLI ({{camel cmd chaos-*}}, {{camel get
chaos}}) and from the MCP tools ({{camel_runtime_chaos_*}}), against a small
app: a timer route {{orders}} with redelivery (2 attempts, 100ms) and a dead
letter channel to {{seda:dlq}}, an HTTP call to a local {{platform-http}} route
{{stock}}, a {{direct}} route with named steps, a {{seda}} hop to {{shipping}}
and a file poller.
Every hook works as documented: HTTP {{status=503}} faults with redelivery and
the DLC, {{step=}} error and delay, {{at=poll drop}}, {{route=}} {{corrupt}} at
the entry, {{stall}} and {{lifecycle action=stop}}, expectations, the shutdown
report, and the parser refuses wrong rules with clear reasons that say what to
use instead. This issue collects what felt wrong or missing while using it, as
input before stage 2 (CAMEL-26607) and the chaos screen (CAMEL-26608).
h3. Usability problems
# *An expectation can be met while every message went to the DLQ.*
{{expect=orders:20}} with {{endpoint=http://...stock* error status=503}} (every
send fails) is *met*: a handled exchange counts as completed (as documented).
The honest check was {{expect=shipping:20}} on the downstream route (FAIL:
"route shipping completed 0 of 20 exchanges within 8s: 20 lost"). Users and AI
agents will write the first form.
# *No expectation on the error handler itself.* "N messages should end in the
DLQ" works only when a route consumes the DLQ ({{expect=dlq:5}} with
{{seda:dlq}} -> route {{dlq}}). It cannot be said for a DLC to {{log:}},
{{mock:}} or {{kafka:dlq}}, and the most common resilience check, "no message
reaches the DLQ" (at most 0), cannot be said at all. And when such an
expectation is unmet, the report says "5 lost", while the 5 messages were in
fact recovered by redelivery: the wording is the opposite of what happened. The
per-route {{handled}} count is already in the report, so see idea 1.
# *A message lost inside the dead letter channel is not reported.*
{{component=seda error until=6}} also failed the send to {{seda:dlq}}; with
{{deadLetterHandleNewException}} (the default) that exchange counts as handled,
but never reached the DLQ ({{orders handled 11}}, {{dlq completed 10}}, a WARN
from DeadLetterChannel). The report says {{lost: 0}} for it. The docs warn
about {{endpoint=*}} and {{component=}} hitting the DLC, but chaos knows the
send is the DLC send (the failure endpoint) and could report it ("r6 failed the
send to the dead letter channel of route orders: 1 message gone"), or skip DLC
sends unless asked for ({{at=dlq}}).
# *A rule that matches nothing gives no feedback.*
{{endpoint=htp://localhost:8765/stock*}} (typo) and {{step=callStok}} were
accepted with only an id, and fired 0 times. The expectation
{{expect=shiping:10}} (no such route) logs a WARN in the application log, but
the CLI and the MCP result only return {{\{"id":"r3"\}}}, so an AI agent never
sees it. Suggest: {{chaos-add}} / {{chaos-change}} return warnings: "matches no
endpoint / step / route of the context now", "expects route shiping, which does
not exist", and the MCP tools pass them through.
# *{{chaos-clear}} does not start a fresh experiment.* After clear the old FAIL
verdict, {{lost}} and the history stay until the application restarts. Suggest
{{chaos-clear --reset}} (or {{chaos-reset}}) that also resets the report, the
route counts and the verdict (a new "session", optionally a new seed).
# *A removed rule with a pending expectation keeps the report not final for
good.* Removing the typo rule {{r3}} (pending expectation) made the report
"FAIL (not final: an expectation is pending)" for the rest of the run. As
documented, but in practice a removed rule's expectation should become
{{cancelled}} (not counted, not pending).
# *"skipped" has no reason.* {{route=shipping corrupt mode=charset}} was
skipped 79 of 79 times: the body after the HTTP call is a stream cache, which
{{charset}} does not change ({{mode=truncate}} worked, the body became
{{in-s}}). Neither the CLI table, the JSON, nor the dev console say why, so it
looks like it worked. Suggest a count per skip reason ({{skipped: \{"stream
body (charset)": 79\}}}), and a hint in the table.
# *Minor:* the JSON report lists the versions of one id as 3, 1, 2; a rule on
{{direct:}} lists {{hooks: ["send","poll"]}} although direct has no polls (the
hooks the target offers, not what applies).
h3. Live rule state (for the TUI chaos screen, CAMEL-26608)
Today a running rule shows {{matches}}, {{fired}} (per hook in JSON),
{{skipped}}, {{ends in}}, and a pending expectation {{completed}} / {{time
left}}. For "fail 5%" that is {{117 matches, 6 fired}}. What a person (and the
chaos screen) needs to see while it runs:
* the observed rate next to the target ({{5.1% of 5%}}), and for {{every:N}}
the next match that fires;
* when it last fired, and a short ring buffer of the last faults: time,
exchange id, route, endpoint/step, fault (so a fault can be matched to a DLQ
entry, a trace or the message history);
* the effect of the rule, not only its faults: of the exchanges it faulted, how
many were recovered by redelivery, handled by the DLC, failed, still in flight.
In the run, {{r7}} (stop {{stock}} for 3s) sent 5 orders to the DLQ, because 2
x 100ms redelivery is too short for a 3s outage: today the report shows
{{orders handled +5}} but not that r7 caused it;
* a rate over time (faults per second over the last minute) for a sparkline;
* mark the exchange: a faulted exchange gets a property (e.g.
{{CamelChaosRule=r1}}) so the tracer, message history, the DLQ route and
{{camel get history}} show "faulted by chaos rule r1".
h3. Ideas for new rules and features
# *Expectations on the error handler:* {{expect=orders:handled<=0}} (redelivery
must recover everything), {{expect=orders:handled=5}},
{{expect=orders:redeliveries<=30}}, {{expect=orders:failed=0}}; the count kinds
already exist per route.
# *Error after the send* ({{at=reply}} or {{after=true}}): the producer is
called, then the send fails, as a timeout after the remote side did the work;
the classic duplicate test for idempotent consumers and retries (a payment that
is retried after it went through).
# *Duplicate:* {{endpoint=kafka:orders* duplicate}} sends the message twice
(at-least-once delivery), to test the idempotent consumer.
# *Burst / outage windows:* {{rate=burst:5s/30s}} fails everything for 5s every
30s, closer to a real outage than an even rate, and what circuit breakers react
to.
# *Slow then fail:* {{delay duration=30s}} combined with an error ({{error
after=30s}}), a timeout that happens after a long wait, which tests the
timeouts of the route.
# *Reply corrupt* at the send: change the reply of the endpoint ({{corrupt
at=reply mode=truncate}}), not only the request.
h3. From reading the code (each traced end to end; 2 and 3 also confirmed on
the running app)
# *A route is counted once per pass, not once per exchange.*
{{ChaosRoutePolicyFactory}} counts in {{onExchangeDone}}, and the route policy
advice runs each time the route processor runs. An exchange that enters a
{{direct}} route 3 times (a {{loop(3).to("direct:sub")}}, or a caller that
redelivers {{to("direct:sub")}} while {{sub}} has {{noErrorHandler()}}) counts
3 completed or failed exchanges in {{sub}}, completes {{expect=sub:30}} with 10
messages, and makes a {{route=sub corrupt}} rule fire and use up its {{until=}}
count 3 times. chaos.adoc says "each counted once when the route is done with
it". Fix: count once per exchange and route (a marker on the exchange), or
document "per pass".
# *{{exception=java.lang.InterruptedException}} bypasses all error handling.*
{{AbstractExchange.setException}} marks the exchange interrupted, so Camel
stops routing it: no {{onException}}, no redelivery, no DLC. Confirmed:
{{step=enrich error exception=java.lang.InterruptedException until=3}} gave 3
failed exchanges in {{orders}}, 0 redeliveries, nothing in the DLQ, only a WARN
of the timer consumer. Fix: reject it (and its subclasses) in the parser, or
wrap it.
# *More rules that can never fire are accepted silently* (adds to problem 4):
{{component=http:localhost}} (a {{component=}} pattern is matched against the
scheme only, so a {{:}} never matches; confirmed, 0 matches), {{rate=every:10
until=5}} (fires 0 times), and {{message=}} with an exception class that has
only a no-argument constructor (the message is dropped). Fix: reject the first
two, reject or warn on the third.
# *Rules from {{camel.chaos.rules}} start their clock when the context
initializes, not when it has started.* {{addedAt}} is set in
{{onContextInitializing}}, so {{until=}}, {{within=}} and the lifecycle times
include the startup. {{expect=orders:100 within=1m}} with routes that take 20s
to start (a slow broker connect, the supervising route controller) leaves 40s.
Fix: re-base the times of the startup rules when the context has started.
# *The report stays not final after a change too* (adds to problem 6): a frozen
expectation of a changed or replaced rule, or one frozen by a context restart,
reads as pending, and {{final}} includes the history, so one {{chaos-change}}
of a rule with an undecided {{expect=}} keeps {{final=false}} for the rest of
the process. An MCP agent loop that waits for {{final}} never ends. Fix: leave
frozen expectations out of {{final}}.
# *The CLI text hides the lifecycle state, and an empty result says nothing.*
{{camel get chaos}} and {{--report}} do not show the {{lifecycle}} object
(down, back in 12s, skipped: route was stopped, left, cancelled): a {{stall}}
row shows only {{active 1 1 0}}, and the reason only appears with {{--json}}.
With no rules, or without camel-chaos on the classpath, {{camel get chaos}}
prints nothing and exits 0. Fix: a LIFECYCLE column (state plus back-in or
reason), and a "no chaos rules" / "camel-chaos is not on the classpath" line.
# *Doc nit:* chaos.adoc says {{\"}}, {{\'}} and {{\\}} are escapes inside
quotes, but only the quote that opened the value and {{\\}} are:
{{message="it\'s"}} keeps the backslash.
More ideas from the code read, on top of the list above: a jittered delay from
the seeded sequence ({{step=enrich delay duration=100ms..2s rate=0.3}}); a
start offset ({{endpoint=kafka:orders* error from=+30s until=1m}}, "the broker
dies after warm-up", which also avoids the startup clock problem); a flapping
route ({{route=poller stall duration=10s repeat=3 every=1m}}, for reconnect,
back-off and supervision); more corrupt modes ({{mode=empty}},
{{mode=remove-header header=JMSCorrelationID}}, {{truncate}} for String and
byte[] bodies, which today are skipped). {{SendHook.afterSend}} is already
there and unused, so the "error after the send" and "corrupt the reply" ideas
fit the SPI as it is; {{duplicate}} needs a "send again" result in the SendHook
SPI, and "slow then fail" needs a delay and an error to combine (today the
first rule that fires decides).
h3. Not in this issue
Route-entry errors, a "sends/steps of route X" filter, {{when=}} conditions and
loose AI prompts are stage 2 (CAMEL-26607). Claus's test prompt from the design
(2026-10-27) "the order route should have chaos, by failing 5% or when a
message origin country is spain" was tried through MCP: {{route=orders error
rate=0.05}} is refused (pointing to endpoint=/step=), the 5% works as
{{endpoint=direct:inventory error rate=0.05}} (6 of 117), and
{{when="$\{header.country\} == 'ES'"}} is refused with "not supported yet". So
the AI needed to look up the first send of the route itself: that is the case
for the stage 2 filter.
_Claude Code on behalf of Claus Ibsen_
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
