Hi Shashank Sorry for the late reply. A late +1 from me as well. Thanks for the clear write-up and posting the numbers.
For anyone reading the archive later, this has landed: - CAMEL-25911 (#28382): TimeSource in camel-util, plus the call sites listed in this thread (redelivery, delayer, throttlers, the task classes, the shutdown wait loop, StopWatch, MonotonicClock). - CAMEL-26103 (#28782): the agent is now loaded with -javaagent instead of a runtime attach. - The send and step hooks (CAMEL-26331, CAMEL-26338) and camel-chaos also read time through TimeSource, so chaos delays run in virtual time too. I have a few follow-ups now that it's in: 1. Your proposal included a forbidden-apis rule, but I don't see one in the build yet. Without it, someone adding a System.nanoTime() or Thread.sleep() to one of these classes would take that wait out of virtual time, and nothing would catch it. A check limited to the classes that were moved would be enough to start with. 2. The javadoc says TimeSource is not an API for components (camel-chaos excepted). The proposal said components would adopt it when they are touched anyway. We should decide which one we mean. If components are expected to use it, it becomes a public API in org.apache.camel.util in practice, and we should treat its compatibility that way. 3. The provider is static, which is fine for tests. In a shared runtime like Karaf, though, one camel-util serves every CamelContext, so an installed provider affects all of them. Logging a line when a non-default provider is installed would make it easy to spot if this ever leaks outside a test harness. 4. +1 to putting it in camel-performance-tests, if that hasn't happened yet. Then anyone can rerun it on x86 or on a new JDK. Looking forward to the next worlds (IMAP, NATS JetStream, Azure Service Bus and Storage Queue, JMS). FYI, there's something similar in Karaf and Camel Karaf coming. Regards JB On Tue, Oct 20, 2026 at 3:08 AM Shashank Jain <[email protected]> wrote: > Hi Claus, > > Thanks for the +1. > > Unless someone has concerns, I will leave the thread open a few more days > and then open a JIRA for it as stage 2 of > CAMEL-25738: the TimeSource in camel-util, the ~30 call sites in core that > change behaviour, and camel-test-dst > installing it, so the rewriting is only needed for the component under > test. > > On more fake worlds: I have worlds for IMAP, NATS JetStream, Azure Service > Bus, Azure Storage Queue and JMS that > found or confirmed bugs this week, and I plan to add them to their > components together with the fixes. > > Looking forward to hearing your other ideas for controlling time once DST > and chaos are in place. > > Thanks, > Shashank > > On Mon, 19 Oct 2026 at 22:53, Claus Ibsen <[email protected]> wrote: > > > Hi Shashank > > > > Thanks for doing these benchmarks so we can see what the affect would be. > > > > Surely the feature of allowing to "control the time" to make > deterministic > > testing in all the complex camel components possible, > > to ensure that we can have a really really well robust component that > > handles all kind of edge cases is worth this feature in the core. > > > > So its a +1 from me. > > > > PS: I have other ideas where this "control the time" can be useful, but > > lets get the DST and the Chaos feature built first. And more fake worlds > in > > various components so we can make Camel very roboust. > > > > > > > > On Mon, Oct 19, 2026 at 1:43 PM Shashank Jain <[email protected]> > > wrote: > > > > > Hi all, > > > > > > Here are the JMH numbers I promised. JMH 1.37, JDK 21.0.11, Apple M4 > Pro > > > laptop (8P+4E cores), camel 4.24.0-SNAPSHOT; > > > 3 forks, 5 x 1 s warmup, 10 x 1 s measurement, average time in ns/op. A > > > laptop result, so indicative only. > > > > > > 1 thread 8 threads > > > System.nanoTime() 8.92 +- 0.01 356.1 +- 3.3 > > > TimeSource.nanoTime(), volatile 9.10 +- 0.06 354.9 +- 3.2 > > > TimeSource.nanoTime(), plain 9.15 +- 0.08 358.0 +- 3.3 > > > TimeSource.nanoTime(), bimorphic 9.22 +- 0.13 355.7 +- 2.8 > > > System.currentTimeMillis() 13.59 +- 0.09 186.5 +- 7.8 > > > TimeSource.currentTimeMillis() 13.62 +- 0.02 185.2 +- 9.5 > > > new StopWatch().taken(), today 19.96 +- 0.04 709.6 +- 7.6 > > > the same with TimeSource 20.52 +- 0.12 710.8 +- 7.8 > > > > > > direct:a -> log:x?level=OFF, route statistics on (today's core, for > > > scale): > > > 2.43M msg/s 10.6M msg/s (8 > > > threads) > > > > > > "bimorphic" means a second provider class was installed and used once > in > > > setup, then SYSTEM was put back, so the > > > call site has seen two receiver types. With 8 threads the clock calls > are > > > much slower on this machine for every > > > variant, System.nanoTime() included; I have not dug into why, but the > > > variants are the same there. > > > > > > What it means: on one thread the indirection costs about 0.2 ns per > > > nanoTime() call and about 0.5 ns per StopWatch, > > > nothing measurable for currentTimeMillis(), and nothing measurable > with 8 > > > threads. The volatile field is not > > > measurably slower than a plain static field here (9.10 vs 9.15 ns), so > I > > > would keep volatile, which also keeps > > > install() safe without extra rules. Against about 410 ns per message > for > > > the simplest direct -> log route, a few > > > tenths of a nanosecond per clock read is well below noise. On x86 a > > > volatile read is also a plain load, but > > > I have only measured on AArch64; numbers from an x86 box would be > > welcome. > > > > > > I can put the benchmark in camel-performance-tests if useful. > > > > > > Thanks, > > > Shashank > > > > > > On Mon, 19 Oct 2026 at 12:44, Shashank Jain <[email protected]> > > > wrote: > > > > > > > > > > > Hi all, > > > > > > > > camel-test-dst (CAMEL-25738, PR > > > https://github.com/apache/camel/pull/28067) > > > > runs the real consumer code of a > > > > component under one seeded scheduler with virtual time, so a test can > > try > > > > thousands of interleavings and faults in > > > > seconds and replay a failing seed exactly. Worlds written with it > found > > > or > > > > confirmed several timing bugs, for > > > > example CAMEL-25683, CAMEL-25737, CAMEL-25741, CAMEL-25756 and > > > CAMEL-25758. > > > > > > > > The harness controls Camel's thread pools through the existing > > > > ThreadPoolFactory SPI. Time is the gap: core reads > > > > System directly, so today a test either rewrites the byte code of > core > > > > classes or lives with real time and real > > > > sleeps in core. Claus suggested a small time source in camel-util, > and > > I > > > > would like to discuss that here before > > > > anything goes into core. > > > > > > > > Where core reads the clock or sleeps today (src/main/java on main, > > > > 2026-10-17): > > > > > > > > camel-util 4 currentTimeMillis, 5 nanoTime, 3 > Thread.sleep > > > > camel-support 24 currentTimeMillis, 6 nanoTime, 6 > > Thread.sleep, > > > 5 > > > > Instant.now > > > > camel-base 43 currentTimeMillis (nearly all event > > timestamps) > > > > camel-base-engine 9 currentTimeMillis, 2 Thread.sleep, 1 > > > > Instant.now, 1 new Date() > > > > camel-core-processor 6 currentTimeMillis, 7 nanoTime, 5 > Thread.sleep > > > > camel-management 9 currentTimeMillis, 1 Thread.sleep, 2 new > > Date() > > > > others (api, languages, console, health, main): a handful > > > > > > > > About 30 of these change behaviour, and only those need to be > > controlled > > > > in a test: > > > > - waits: RedeliveryPolicy.sleep and the sleep loop of > > > > RedeliveryErrorHandler, ForegroundTask, the wait loop of > > > > DefaultShutdownStrategy, DelayProcessorSupport, MulticastProcessor, > > > > OptimisticLockRetryPolicy, > > > > DefaultStartupConditionStrategy; > > > > - task timing: StopWatch, MonotonicClock, AbstractTask, > BackgroundTask, > > > > TimeBoundedBudget, BackOffTimerTask, > > > > DefaultTimeoutMap; > > > > - throttling: TotalRequestsThrottler, ConcurrentRequestsThrottler, > > > > SamplingThrottler, > > > > ThrottlingExceptionRoutePolicy. > > > > > > > > Event timestamps, the debugger, the tracer, management and the > consoles > > > > can stay on System. > > > > > > > > Options I see: > > > > > > > > 1. Byte code rewriting only (what stage 1 does). No core change, but > > > > runtime agent attach is being phased out in the > > > > JDK, it only covers the calls it is told about, and rewritten core > > > > classes stay rewritten for the whole test JVM. > > > > > > > > 2. A time source for all of core: about 120 call sites, many on hot > > > paths, > > > > for no gain in tests. > > > > > > > > 3. In between, which is what I would propose: > > > > - a small static, internal TimeSource in camel-util: > > > > currentTimeMillis(), nanoTime(), instant(), sleep(long). > > > > The default delegates to System and Thread.sleep; a test harness > > can > > > > install another provider. Static rather > > > > than per CamelContext, because StopWatch and MonotonicClock have > > no > > > > context; > > > > - move only the ~30 call sites above; > > > > - keep byte code rewriting for the component under test, pointed > at > > > > TimeSource, with the agent loaded by > > > > -javaagent instead of attached at runtime; > > > > - later, components use it when they are touched anyway, and a > > > > forbidden-apis rule keeps new > > > > Instant.now() / Thread.sleep() out of consumers. > > > > > > > > Rough shape: > > > > > > > > public final class TimeSource { > > > > public interface Provider { > > > > long currentTimeMillis(); > > > > long nanoTime(); > > > > void sleep(long millis) throws InterruptedException; > > > > } > > > > private static volatile Provider provider = SYSTEM; > > > > public static long nanoTime() { return provider.nanoTime(); } > > > > ... > > > > public static Provider install(Provider p) { ... } // test > > > > harnesses only > > > > } > > > > > > > > Open points: package and name; a volatile field or a plain static > field > > > > set once before the context starts; and > > > > whether sleep belongs there or in a separate wait helper. > > > > > > > > On cost: I will run a JMH benchmark (direct System.nanoTime vs. > > > TimeSource > > > > with a volatile and a plain field, > > > > StopWatch, and a direct -> log route with statistics on) and post the > > > > numbers in this thread. My expectation is > > > > that the default path inlines to the same call plus one ordinary > load, > > so > > > > it should be within noise. > > > > > > > > Thoughts? > > > > > > > > Thanks, > > > > Shashank > > > > > > > > > > > > > -- > > Claus Ibsen > > >
