Claus Ibsen created CAMEL-26701:
-----------------------------------
Summary: camel-chaos - a fault caught by a circuit breaker
fallback counts as failed, and a recovered fault marks later errors
Key: CAMEL-26701
URL: https://issues.apache.org/jira/browse/CAMEL-26701
Project: Camel
Issue Type: Bug
Components: camel-jbang
Reporter: Claus Ibsen
Fix For: 4.24.0
Two bugs from a hands-on round of the chaos screen (CAMEL-26699), with a demo
shop that calls a payment service behind a resilience4j circuit breaker that
has a fallback.
h3. 1. A fault caught by a circuit breaker fallback counts as failed
The rule {{endpoint=http://localhost:8765/pay* error
exception=java.net.ConnectException rate=burst:10s/30s}} reported its effect as
{{156 failed}}, while the payment route had 0 failed and every order was
shipped through the fallback ("payment later").
The circuit breaker runs its steps on a correlated copy of the exchange, with a
unit of work of its own ({{ResilienceProcessor.processTask}}). Chaos faults
that copy, and counts the effect when the unit of work of the copy is done: the
copy failed, so FAILED. The exchange itself then goes on with the fallback, or
with the exception when there is no fallback, and its outcome is never counted.
Fix: the faults of a circuit breaker copy that failed are counted when the
exchange it is a copy of is done, with that exchange's outcome: completed (the
fallback took over), handled by the error handler, or failed. A circuit breaker
copy is a correlated copy inside a try block ({{TRY_ROUTE_BLOCK}}, which the
breaker sets) with a unit of work in the same route. The copies of a splitter,
a multicast or a wire tap are not affected, and still count on their own. The
original exchange is not touched from the thread of the copy: the faults wait
under its exchange id until its completed or failed event, which the event
notifier listens for only while some wait.
h3. 2. A real error is marked as made by chaos
The exchange property {{CamelChaosRule}} stays on an exchange after the fault
was recovered by a redelivery. So a later failure of the same exchange that
chaos did not make shows the mark of the old rule on the Errors tab. In the
demo, a {{CallNotPermittedException}} of the open circuit breaker was marked
{{🎲r1}}, the rule on the stock call, which had been recovered.
Fix: when the error handler redelivers the exchange, the property is removed,
as the faulted attempt is retried; a fault at the redelivery sets it again. The
Errors tab of the terminal UI trusts the properties of an error when it has
them, and only falls back to the recent faults of the report when it has none.
_Claude Code on behalf of Claus Ibsen_
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
