allthingssecurity opened a new pull request, #28812:
URL: https://github.com/apache/camel/pull/28812
# Description
CAMEL-26456 (related to CAMEL-26330 and CAMEL-26418): **route lifecycle
faults**, step 6 of "Order of work" in
`design/chaos.adoc`. Step 8 (a DST world that runs chaos rules in virtual
time) is a stacked follow-up PR on top of this
one; steps 3 (the dev console and the CLI, #28788) and 7 (the MCP tool
group) are in their own PRs. It is built on main and does not need #28788: the
new rules go through `camel.chaos.rules`
and `ChaosManager`, so the CLI and the MCP tools take them as they take any
rule (the console JSON carries the new
`lifecycle` object; see "After #28788" below).
The rule syntax of the design already had these faults, and the parser
rejected them as "not supported yet":
```
route=poller stall duration=20s # suspend, resume
after 20 s
route=orders lifecycle action=suspend duration=1m # the same
route=orders lifecycle action=stop duration=1m # stop, start after 1
m
route=orders lifecycle action=remove duration=10s # remove, add the
same route definition back after 10 s
```
## What a lifecycle rule does
- **Once, when the rule is added**, as the design says (a rule of
`camel.chaos.rules` when the context has started).
`at`, `rate` and `until` are rejected (`rate= does not apply to the fault
stall: it acts once on the route when the
rule is added, and brings it back after its duration`), and the `duration`
is at most `camel.chaos.maxDuration`. An
expectation (`expect=orders:100 within=5m`) can be added.
- **The route is checked by the same parser**: `route=` takes the id of one
route (a pattern is rejected), which must
exist when the rule is added (`there is no route nope (the fault stall
acts on a route that exists when the rule is
added)`). A rule of `camel.chaos.rules` is checked when the context starts
(`onContextStarting`, with its routes
created), and an unknown route stops the context from starting, as any
rule that is not valid does. While a rule has
a route removed, a rule for that route is still accepted (to change the
rule).
- **On the scheduled executor of chaos**: the action and the comeback are
tasks of one
`newSingleThreadScheduledExecutor` of the `ExecutorServiceManager`,
created when the first lifecycle rule acts. So
adding a rule returns at once (a suspend or stop can wait for the shutdown
strategy), the tasks run one after another,
and in a camel-test-dst world they run on the loop of the run. The route
is suspended, stopped, resumed and started
through `camelContext.getRouteController()`, and removed with
`CamelContext.removeRoute`.
- **`action=remove` adds the same route definition back** with
`Model.addRouteDefinition`: the definition that was
removed, as it was prepared, so the route comes back with the route
configurations it had. That needs camel-core-model
as a dependency of camel-chaos (as camel-kamelet and
camel-observability-services have; nothing in camel-api adds a
definition). A route created from a route template or a kamelet is
rejected for `remove`: removing the route closes
the local beans of its template (`LocalBeanRegistry`), which the
definition still refers to, so it could not come
back. `stop` and `stall` work for it.
## Chaos only brings back what it took down
- A route that is not started when the rule acts is left alone (state
`skipped`, "route a was stopped").
- At the comeback, a route that someone else changed meanwhile (stopped a
suspended route, removed it, or a reload in
dev mode loaded a new route with the same id) is left as it is (`left`,
with the reason).
- Removing, changing or clearing the rule while the route is down brings it
back at once (`endedEarly`). A changed rule
acts after the old version brought the route back (not "skipped" because
chaos itself had it down).
- When the context stops, a pending comeback is cancelled and the executor
shut down, so nothing of chaos keeps running
(`cancelled`, "the context stopped, so route a was not resumed"). A
removed route then stays removed when the context
is started again; the docs say so.
- A failed comeback (a route that fails to start) is reported `failed`. An
action that fails halfway (a remove that
could stop the route but not remove it) brings the route back as it left
it, after the duration (`down`, then
`back`, with the reason).
- **Supervising route controller**: a stop or suspend through the route
controller is a manual operation for
`DefaultSupervisingRouteController` (it releases the route:
`route.getRouteController()` is null while down), and the
comeback starts the route through it, which supervises it again (tests pin
both). A route that `remove` adds back is
taken under supervision by the controller but not started by it, so chaos
starts it through the controller (a test
failed before that). Since that controller starts routes in the
background, a rule of `camel.chaos.rules` can find its
route not started yet and does nothing; the docs say so.
- **The report** keeps what each lifecycle rule did, with the times after
the rule was added (read through
`TimeSource`):
```
r2 ended: route=orders lifecycle action=suspend duration=60000ms
expect=orders:8 within=50m
matches 1, fired 1, skipped 0
suspended route orders at +0ms, resumed at +1m0s (down 1m0s)
expect orders:8 within 50m0s: met, 8 completed
```
In JSON the action counts as `fired.lifecycle`, and the rule has a
`lifecycle` object: `action`, `route`, `duration`,
`state`, `downAt`, `backAt`, `backIn` while down, `endedEarly`, `reason`.
A lifecycle rule has no `endsIn` (it ends
when its route is back), and it is `ended` then. The report keeps removed
rules as before, with their lifecycle state.
- **Fail closed** as everything else: with profile `prod`, or without the
profile `dev` or `camel.chaos.enabled=true`,
nothing of chaos is installed, so these rules are ignored (a test pins it
for a lifecycle rule in `camel.chaos.rules`).
## Docs
- The chaos page: the faults table (`lifecycle`, `stall`), the targets
table, a new section "Route lifecycle faults"
(all of the above), and "Time and the seed". The catalog copy is
regenerated.
- `design/chaos.adoc`: the lifecycle row says the target is one route that
exists, and what chaos does not bring back;
"Order of work" marks the steps that are done (1, 2, 4, 5, and 6 with this
PR). `last-updated` is left alone, as
#28788 changes that line.
- No upgrade guide entry: no existing option or rule changes behaviour
(`stall` and `lifecycle` were rejected before).
## Tests
New test classes (each run 20 times in its own JVM, all passed):
- `ChaosLifecycleTest` (19): stall, stop and remove with their comeback (the
same definition added back, with one chaos
route policy, counting an exchange once); the route takes messages again;
a route that is not started is left
alone; a route stopped by someone while suspended, a route loaded again
and a route replaced meanwhile are left as
they are; removing the rule brings the route back at once, also while the
action runs (a stop waiting for an exchange
in flight); a rule changed while it acts acts after the old version
brought the route back; change and clear end the
old fault; a remove that fails after the stop starts the route after the
duration; a comeback that fails is reported;
a route from a route template is not removed but can be stopped; a context
stop cancels the comeback (no executor, no
`Chaos` thread left); rules act again after the context is started again;
an expectation of a stalled route; the
route checks (unknown route, max duration, `rate=`; a rejected rule takes
no id).
- `ChaosLifecycleStartupTest` (9): a rule of `camel.chaos.rules` acts when
the context has started and waits while it
is not; removed before the start, it does nothing; an unknown route or
`until=` stops the context from starting;
ignored with profile `prod`; the supervising route controller (stop:
released while down, supervised again after;
stall; remove: added back, started and supervised).
- `ChaosRuleParserTest`: the forms of `stall` and `lifecycle`, every
rejection with its message, and `action=` on
another fault; the fuzz test accepts a lifecycle rule without hooks.
The camel-chaos module passes (303 tests). Negative controls: 21 mutants of
the new code, each compiled into the
module and run against its test classes; 20 make a test fail (no comeback, a
comeback at once or 1 ms early, acting on
a stopped route, no "left" check, replacing a reloaded route, removing the
rule keeps the route down, the context stop
not told, a suspend that does nothing, `rate=`, a pattern or a too long
duration accepted, the route not checked at add
or at start, startup rules that never act, a changed rule not deferred, a
re-added route not started under supervision,
a stop that bypasses the route controller, a halfway failure left down, the
report without `lifecycle`). One survives
as equivalent: not shutting the executor down at stop, since the
`ExecutorServiceManager` shuts its executors down when
the context stops.
## Merge with the open chaos PRs
`git merge-tree` against #28788 (`camel-chaos-control`) is clean. Against
CAMEL-26432 (`camel-chaos-netty-http-status`)
one line conflicts, in the table of rejected rules of `ChaosRuleParserTest`:
this PR changes the expected message of
`endpoint=x error action=stop` and that PR the line below it; whichever
merges second keeps both lines.
## After #28788
The `chaos` console of #28788 shows the report JSON, which then has the
`lifecycle` object; its `RuleEntry` metadata
and `camel get chaos` (the ENDS IN column is empty for a lifecycle rule) do
not know it yet. I can add that in whichever
of the two merges second, or as a follow-up JIRA.
_Claude Code on behalf of allthingssecurity_
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
