allthingssecurity opened a new pull request, #30243:
URL: https://github.com/apache/camel/pull/30243

   # Description
   
   [CAMEL-26966](https://issues.apache.org/jira/browse/CAMEL-26966), part of 
[CAMEL-26630](https://issues.apache.org/jira/browse/CAMEL-26630) (exchange 
sanitizer), after R2 
([CAMEL-26768](https://issues.apache.org/jira/browse/CAMEL-26768), #29664).
   
   It builds on the dev console and `camel get sanitizer` of 
[CAMEL-26769](https://issues.apache.org/jira/browse/CAMEL-26769) (#29901) and 
needs [CAMEL-26965](https://issues.apache.org/jira/browse/CAMEL-26965) (#30129, 
`interceptSendToEndpoint`), both merged: the camel-core run below found that 
one processor of Camel broke the rule, and it got its own fix first, as the 
design asks, since R3 is on by default in tests (see "Validation").
   
   Step 4 of the "Order of work" of `design/exchange-sanitizer.adoc`, first 
part: the `AsyncProcessor` contract of rule R3 ("one thread at a time"). The 
second part of R3, the sampled watch windows, is not in this PR. R1 and R2 are 
unchanged.
   
   ## What it adds
   
   The rule, as the Javadoc of `AsyncProcessor` states it: the value the 
callback of a call is called with (`doneSync`) is the value `process(exchange, 
callback)` returns, and `true` means the callback was called already, on the 
calling thread. A caller that uses the return value continues the exchange on 
`true` and leaves it to the callback on `false`, so a processor that returns 
the other value lets two threads continue the exchange, or none. Three new 
kinds of `Violation` (`getRule()` returns `R3`), checked at the call of each 
step, below its error handler, as R1 is:
   
   | Kind | When |
   |---|---|
   | `RETURNED_TRUE_CALLBACK_ON_OTHER_THREAD` | `process` returned `true` and 
its callback had been called on another thread (such as a processor that waits 
for its worker) |
   | `RETURNED_TRUE_BEFORE_CALLBACK` | `process` returned `true`, and the 
callback came later on another thread, or with `true` |
   | `RETURNED_FALSE_CALLBACK_TRUE` | `process` returned `false` and its 
callback was called with `true` (either order, any thread) |
   
   ```text
   Exchange sanitizer: process returned true, and its callback was called on 
another thread (R3)
     exchange 1FF90A20897EAF2-0000000000000000, route prices, step lookup, 
processor org.apache.camel.sanitizer.BrokenUserCode$ReturnsTrueAfterAsyncDone
     the callback was called with false: thread user-worker
         at 
org.apache.camel.sanitizer.BrokenUserCode$ReturnsTrueAfterAsyncDone.lambda$process$0(BrokenUserCode.java:180)
         at java.base/java.lang.Thread.run(Thread.java:1583)
     then process returned true: thread Camel (camel-61) thread #91 - 
ProducerTemplate
         at org.apache.camel.support.StepHelper.process(StepHelper.java:63)
         at 
org.apache.camel.processor.errorhandler.RedeliveryErrorHandler.processOutput(RedeliveryErrorHandler.java:266)
         ...
     process(exchange, callback) must return the value its callback is called 
with (see AsyncProcessor).
     Usual fix: when the work goes on another thread, return false from process 
and let that thread call the callback with false; return true only after the 
callback was called with true on the calling thread.
   ```
   
   - **`SanitizerDoneSyncListener`**, a second `StepListener` with a small 
record per call (`DoneSyncCall`). It needs no table and keeps no exchange or 
thread (the id of the calling thread only). The return and the first callback 
are applied one at a time under the lock of the record, and whichever comes 
second compares, once. A stack is taken only for a callback on another thread 
before the return, or when a violation is reported.
   - **Option.** `camel.sanitizer.checks` gets `threads`. The default becomes 
`callback,threads`, and `callback,commit,threads` in a test, as the options 
table of the design lists `threads` in the default and R3a costs little (see 
Cost). camel-main option, generated metadata, `main.adoc`, catalog copies. 
`checks=threads` alone installs only this listener, which then counts the calls.
   - **Console and CLI.** The `sanitizer` dev console gets a `checks` field 
(the checks that run) and its descriptions name R3; `camel get sanitizer 
--report` says "R3 not checked" when `threads` is not in the checks, and does 
not suggest `threads` to an integration of an older version (it would not start 
with an unknown check). The rest of the console and the command show the new 
kinds as they are.
   - **Docs.** An R3 section in `sanitizer.adoc` (the kinds, the case that is 
not reported and why, a report); the design (R3 narrowed, step 4 partly built, 
the defaults); the `--sanitizer` text of `camel run`, `camel dev` and `camel 
debug` and the jbang managing page say `callback,commit,threads` where they 
told how to turn R2 on, so following them does not turn R3 off.
   
   ## Where this differs from the design
   
   **Validation of R3 is done for this part on camel-core only.** The design 
asks, before R3 is complete, for the runs on the async component suites (netty, 
netty-http, vertx-http, JMS request/reply), a TLA+ model of the record, and the 
JMH run. They come with the sampled overlaps; the design says so now. The 
record of this part is two values compared under one lock, and the tests drive 
every order and race it. The cost below is a quick loop, not JMH.
   
   **Narrowed for the case the Javadoc allows.** A processor that hands its 
callback to the reactive executor returns the value of the processor it calls, 
and the reactive executor calls the callback with `false` on the calling 
thread, at once or after the work that thread is running. The channel of a step 
without an error handler does this, so a step that calls such a channel 
directly sees it, such as a `choice` whose branch has one step in a route with 
`noErrorHandler()` (with the default error handler the channel returns 
`false`). So a callback with `false` on the thread that called `process` is 
never reported, whatever `process` returns and whether it comes before or after 
the return. This relies on the default reactive executor of camel-core, which 
runs a callback on the thread that scheduled it. The design text says so. A 
side effect: a processor that hands its work to a pool that the calling thread 
also belongs to, and gets the same thread back for the callback, is not seen.
   
   ## Validation
   
   **1. User code.** `SanitizerDoneSyncTest` (camel-core, `@CamelSanitizer`): a 
processor that waits for its worker and returns `true` while the worker called 
the callback with `false` (the planted "two threads continue the exchange"); 
one that returns `true` and lets a worker call back around the return; one that 
calls back with `true` from a worker and returns `false`; one that calls back 
with `true` and then returns `false`. Each fails its test with a report that 
names its step, route and class, unless it says `@ExpectSanitizerViolation`. 
200 broken calls at once give exactly 200 reports. Correct code is quiet: an 
asynchronous processor on a pool, `to("direct:...")`, a `choice` in a route 
with `noErrorHandler()` (where a recording listener checks that the allowed 
case really happened: returned `true`, callback with `false` on the calling 
thread), `threads()`, a parallel `split`, `delay().asyncDelayed()` and 
`to("seda:...")`; the asynchronous processor, the `direct` route on a pool
  and the EIPs also 200 times at once.
   
   **2. The record.** `SanitizerDoneSyncCallTest` drives `DoneSyncCall` in 
every order of the return, the throw and the first and second callbacks, on the 
calling thread and on another one, and races the return against a callback on 
another thread 2000 times for three pairs of values (returned `true` with a 
callback with `false`, `true` with `true`, `false` with `true`) and for the 
correct pair (`false` with `false`): one report for each broken call, none for 
the correct one. Negative controls, each a mutant of the listener run against 
`SanitizerDoneSyncTest` and `SanitizerDoneSyncCallTest` (18 tests): no 
exception for the reactive executor, 2 tests fail (also the route with the 
`noErrorHandler()` choice); no report at the return of a callback that came 
first on another thread, 5 fail; no compare at the callback, 6 fail; no check 
of a throw, 1 fails. Without the lock nothing failed in 10 runs of the race 
test: the two threads never lost a compare, so the tests cannot show what the lo
 ck is for.
   
   **3. Camel on camel-core.** The whole camel-core suite with the CI switch 
(`-Psanitizer`): 9368 tests, 26 failures, 12 errors. R3 reports nothing in 
Camel's EIPs except one processor: **`InterceptSendToEndpointProcessor`** 
returns the value of the `before` of the interceptor (a route, so `false`) and 
calls the callback with the value of the producer after it (`true` for `mock`), 
and with `mockEndpoints()` and an asynchronous producer the other way round. 
That fails 16 test classes that use `interceptSendToEndpoint` or advice with 
`mockEndpoints` (37 reports, all at that processor). It is a real break of the 
contract: a caller that uses the return value would continue the exchange twice 
or not at all (Camel's own callers go on through the callback, so the routes 
work today). It is fixed in CAMEL-26965 (#30129); with that fix applied the 
same run (9371 tests) has no R3 report outside the tests that break the rule on 
purpose; what fails there is the list below, plus one case of the f
 ix's own new test that I corrected and ran again on its own. The other 
failures of the run are not R3: `AggregateOnCompletionThrowsTest` (2, an R1 
report of a test added by CAMEL-26881 whose strategy throws an `Error` on 
purpose; it needs `@ExpectSanitizerViolation`, as #29672 did for similar 
tests), the `DumpModel*` tests (a stale `camel-yaml-io` in my local repository) 
and `ValidatorExternalResourceTest` (network), which fail the same way without 
the sanitizer.
   
   **4. Tests.** The sanitizer tests of camel-core, camel-management, 
camel-main and camel-test-main-junit6, `StepListener*Test` (camel-core 178, 
camel-management 1, camel-main 10, camel-test-main-junit6 4), 
`ChaosWithSanitizerTest` 6, the cli-connector sanitizer tests 3, 
`ProjectRunnerTest` 21, and in camel-jbang-core `ListSanitizerTest`, 
`ListSanitizerActionTest`, `ListChaosTest`, `RunTest`, `CommandInputErrorsTest` 
and the CLI help and parse goldens, 96, all pass. `RunExportGoldenTest` fails 
18 of 49 cases here: those cases resolve dependencies that my offline build 
does not have, and they do not touch the sanitizer. 20 times in one JVM, 
`SanitizerDoneSyncTest`, `SanitizerDoneSyncCallTest`, `SanitizerActivationTest` 
and `SanitizerUserCodeTest` and `SanitizerDevConsoleTest`: 1180 of 1180; with 
the CI switch, `SanitizerDoneSync*`, `StepListenerTest`, `ChoiceTest` and 
`AsyncEndpointTest`: 700 of 700.
   
   ## Cost
   
   Off (no `threads` in the checks, or no sanitizer): nothing is installed and 
nothing runs. On: one small object per call of a step, two short uncontended 
locks, two reads of the current thread, and a stack only for a callback on 
another thread before the return. A quick loop (not JMH; the 6 steps of the R2 
benchmark, `pendingTimeout=0`, one sending thread, 5 rounds of 200 000 messages 
after a warm-up), per message: sanitizer off 1.00 us; R1 1.21 to 1.28 us; R1 
and R3 1.25 to 1.30 us; R3 alone 1.21 to 1.27 us (`stackDepth=0`). With the 
default 32 frames, R1 is 53.5 to 54.2 us with or without R3: R1 takes a stack 
at every callback, and R3 is in the noise. The harness is outside this PR; I 
can add a case to the sanitizer JMH benchmark of camel-performance-tests if 
wanted.
   
   ## Follow-ups (not in this PR)
   
   - The second part of R3, the sampled watch windows, and the runs of R3 on 
the test suites of netty, netty-http, vertx-http and JMS request/reply, as the 
design asks before R3 is complete. This PR ran camel-core only.
   - `AggregateOnCompletionThrowsTest` with `@ExpectSanitizerViolation`.
   
   # Target
   
   - [x] I checked that the commit is targeting the correct branch (Camel 4 
uses the `main` branch)
   
   # Tracking
   - [x] If this is a large change, bug fix, or code improvement, I checked 
there is a [JIRA issue](https://issues.apache.org/jira/browse/CAMEL) filed for 
the change (usually before you start working on it): 
[CAMEL-26966](https://issues.apache.org/jira/browse/CAMEL-26966).
   
   # Apache Camel coding standards and style
   
   - [x] I checked that each commit in the pull request has a meaningful 
subject line and body.
   - [ ] I have run `mvn clean install -DskipTests` locally from root folder 
and I have committed all auto-generated changes.
     (I built the core modules, the base components, camel-sanitizer, 
camel-management, camel-main, camel-test-main-junit6, camel-cli-connector and 
camel-jbang-core with the formatter and import-sort plugins and the package 
plugin; the camel-main metadata and `main.adoc`, the dev console JSON and the 
camel-jbang command metadata and pages are generated, and the catalog copies 
match them. I did not run the full root build.)
   
   # AI-assisted contributions
   
   - [x] If this PR includes AI-generated code, commits have proper 
co-authorship attribution (e.g., `Co-authored-by` trailers) and the PR 
description identifies the AI tool used.
     This PR was prepared with Claude Code (Claude Opus 5.5). The commit 
carries a `Co-Authored-By` trailer.
   
   _Claude Code on behalf of allthingssecurity_
   
   🤖 Generated with [Claude Code](https://claude.com/claude-code)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to