This is an automated email from the ASF dual-hosted git repository.

davsclaus pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/camel.git


The following commit(s) were added to refs/heads/main by this push:
     new 184121cf5bca CAMEL-26956: camel-azure-servicebus - a DST fake world 
for the Service Bus consumer (#30130)
184121cf5bca is described below

commit 184121cf5bca9fdaefc32a6bd6ca62d3ff09da10
Author: allthingssecurity <[email protected]>
AuthorDate: Tue Nov 17 14:28:45 2026 +0530

    CAMEL-26956: camel-azure-servicebus - a DST fake world for the Service Bus 
consumer (#30130)
    
    …e Bus consumer
    
    Adds a camel-test-dst world for the camel-azure-servicebus consumer in
    peek-lock mode, in the dst package of its tests: ServiceBusConsumerDstTest
    (1000 seeds), ServiceBusWorld and FakeServiceBus. The real consumer and its
    lock renewer run against a fake queue behind the processor and receiver
    clients of a mocked ServiceBusClientFactory set on the endpoint, written
    from the Service Bus documentation on locks, settlement and dead-lettering
    and from azure-messaging-servicebus 7.17: locks for the lock duration of the
    queue, redelivery and a higher delivery count after an expiry or an abandon,
    dead-lettering past MaxDeliveryCount, complete, abandon and dead-letter that
    fail once the lock is lost, renewals that take a while and fail when they
    come too late, a processor whose stop() closes its receiver (the V2 stack,
    the default of the SDK), after which a settlement throws, and the lock
    renewal of the processor client while a processMessage call runs (up to
    maxAutoLockRenewDuration, ended when the call returns or the receiver is
    closed).
    
    The route has an asynchronous step, so only the lock renewer of the
    consumer keeps the locks of the routed messages. The seed picks the lock
    duration, the max delivery count, maxConcurrentCalls,
    maxAutoLockRenewDuration (60 or 300 seconds), enableDeadLettering and the
    fault rates, and injects failed and slow renewals, failed settlements,
    failed processing, processing longer than the lock duration, and route
    restarts while messages are processed. The operator stops the route as
    DefaultShutdownStrategy does, with the wait of the strategy as tasks of
    the operator, so that a processMessage call that waits until the consumer
    stops does not sit above the stop on the one stack; a stop that times out
    forces the consumer to stop, and the exchanges still in flight fail;
    their settlements then find the client closed (by design), so
    strictClose counts only the settlements inside the graceful wait.
    
    Invariants: no lock is lost and no message is delivered again while the
    consumer holds it (a routed delivery whose exchange is not done), unless a
    renewal failed, the route was stopped or maxAutoLockRenewDuration passed;
    the message of a completed exchange is completed, unless its settlement
    failed or was excused; a message is completed only after its exchange
    completed, and abandoned or dead-lettered only after it failed (or was
    in flight when a stop timed out); once the faults cleared every message is 
completed or dead-lettered; the consumer
    settles a delivery at most once and renews only what it holds, through an
    open client, and routes or settles a delivered message before
    processMessage returns, unless the route is being stopped.
    
    Two behaviours are accepted by default and counted with a property: the
    consumer stops renewing a message after one renewal error
    (-Ddst.servicebus.strictRenew=true, 28 of 1000 seeds fail; CAMEL-26969),
    and the settlement of an exchange that completes during the graceful stop
    of the route, after the stop() of the processor closed its receiver,
    fails, so the message is delivered again
    (-Ddst.servicebus.strictClose=true, 96 of 1000 seeds fail; CAMEL-26970).
    
    Self-check: on the code before CAMEL-25701 (4f879148b118 reverted) 76 of
    2000 seeds fail; main passes 5000 of 5000 seeds.
    
    The module's pom loads the byte code agent of camel-test-dst, the test log
    is capped, and the catalog lists the world.
    
    Co-Authored-By: Claude Opus 5.5 <[email protected]>
---
 .../camel/catalog/components/azure-servicebus.json |   3 +
 .../org/apache/camel/catalog/docs/test-dst.adoc    |  10 +
 .../camel-azure/camel-azure-servicebus/pom.xml     |  26 +
 .../azure/servicebus/azure-servicebus.json         |   3 +
 .../azure/servicebus/dst/FakeServiceBus.java       | 467 +++++++++++++++++
 .../servicebus/dst/ServiceBusConsumerDstTest.java  |  37 ++
 .../azure/servicebus/dst/ServiceBusWorld.java      | 582 +++++++++++++++++++++
 .../src/test/resources/log4j2.properties           |   9 +-
 .../camel-test-dst/src/main/docs/test-dst.adoc     |  10 +
 9 files changed, 1146 insertions(+), 1 deletion(-)

diff --git 
a/catalog/camel-catalog/src/generated/resources/org/apache/camel/catalog/components/azure-servicebus.json
 
b/catalog/camel-catalog/src/generated/resources/org/apache/camel/catalog/components/azure-servicebus.json
index a79da4aee71f..1c868480c984 100644
--- 
a/catalog/camel-catalog/src/generated/resources/org/apache/camel/catalog/components/azure-servicebus.json
+++ 
b/catalog/camel-catalog/src/generated/resources/org/apache/camel/catalog/components/azure-servicebus.json
@@ -119,5 +119,8 @@
     "credentialType": { "index": 28, "kind": "parameter", "displayName": 
"Credential Type", "group": "security", "label": "security", "required": false, 
"type": "enum", "javaType": 
"org.apache.camel.component.azure.common.CredentialType", "enum": [ 
"AZURE_IDENTITY", "CONNECTION_STRING", "TOKEN_CREDENTIAL" ], "deprecated": 
false, "autowired": false, "secret": false, "defaultValue": 
"CONNECTION_STRING", "configurationClass": 
"org.apache.camel.component.azure.servicebus.ServiceBusConfigurat [...]
     "fullyQualifiedNamespace": { "index": 29, "kind": "parameter", 
"displayName": "Fully Qualified Namespace", "group": "security", "label": 
"security", "required": false, "type": "string", "javaType": 
"java.lang.String", "deprecated": false, "autowired": false, "secret": false, 
"configurationClass": 
"org.apache.camel.component.azure.servicebus.ServiceBusConfiguration", 
"configurationField": "configuration", "description": "Fully Qualified 
Namespace of the service bus" },
     "tokenCredential": { "index": 30, "kind": "parameter", "displayName": 
"Token Credential", "group": "security", "label": "security", "required": 
false, "type": "object", "javaType": 
"com.azure.core.credential.TokenCredential", "deprecated": false, "autowired": 
true, "secret": true, "security": "secret", "configurationClass": 
"org.apache.camel.component.azure.servicebus.ServiceBusConfiguration", 
"configurationField": "configuration", "description": "A TokenCredential for 
Azure AD authe [...]
+  },
+  "dstWorlds": {
+    "ServiceBusWorld": { "javaType": 
"org.apache.camel.component.azure.servicebus.dst.ServiceBusWorld", "fake": 
"FakeServiceBus: a queue in peek-lock mode (lock duration, delivery count, 
dead-letter queue) behind the processor and receiver clients of 
ServiceBusClientFactory", "covers": [ "the lock renewer of the consumer 
(PEEK_LOCK, maxAutoLockRenewDuration) with an asynchronous route", "processing 
that takes longer than the lock duration", "complete, abandon and dead-letter, 
also after  [...]
   }
 }
diff --git 
a/catalog/camel-catalog/src/generated/resources/org/apache/camel/catalog/docs/test-dst.adoc
 
b/catalog/camel-catalog/src/generated/resources/org/apache/camel/catalog/docs/test-dst.adoc
index 43249aaa404e..c834da4f739a 100644
--- 
a/catalog/camel-catalog/src/generated/resources/org/apache/camel/catalog/docs/test-dst.adoc
+++ 
b/catalog/camel-catalog/src/generated/resources/org/apache/camel/catalog/docs/test-dst.adoc
@@ -91,6 +91,16 @@ a|* checkpointing by checkpointBatchSize and 
checkpointBatchTimeout
 * an asynchronous downstream step
 |CAMEL-25683, CAMEL-25738, CAMEL-25781
 
+|camel-azure-servicebus
+|`ServiceBusWorld` (`ServiceBusConsumerDstTest`)
+|FakeServiceBus: a queue in peek-lock mode (lock duration, delivery count, 
dead-letter queue) behind the processor and receiver clients of 
ServiceBusClientFactory
+a|* the lock renewer of the consumer (PEEK_LOCK, maxAutoLockRenewDuration) 
with an asynchronous route
+* processing that takes longer than the lock duration
+* complete, abandon and dead-letter, also after the lock was lost
+* maxConcurrentCalls of one to three (with the asynchronous route it bounds 
the processMessage calls)
+* route stop and start while messages are processed
+|CAMEL-25701, CAMEL-26956, CAMEL-26969, CAMEL-26970, CAMEL-27066
+
 |camel-google-pubsub
 |`PubsubWorld` (`PubsubConsumerDstTest`)
 |FakePubsub: a subscription (Pull, Acknowledge, ModifyAckDeadline, ack 
deadlines, dead letter policy) behind the SubscriberStub of synchronousPull
diff --git a/components/camel-azure/camel-azure-servicebus/pom.xml 
b/components/camel-azure/camel-azure-servicebus/pom.xml
index 9687ce9b3207..08eac037e281 100644
--- a/components/camel-azure/camel-azure-servicebus/pom.xml
+++ b/components/camel-azure/camel-azure-servicebus/pom.xml
@@ -36,6 +36,8 @@
         <!-- Floci AZ is not available on these platforms -->
         <skipITs.ppc64le>true</skipITs.ppc64le>
         <skipITs.s390x>true</skipITs.s390x>
+        <!-- the DST tests load the byte code agent of camel-test-dst when the 
JVM starts (see test-dst.adoc) -->
+        
<camel.surefire.fork.additional-vmargs>"-javaagent:${net.bytebuddy:byte-buddy-agent:jar}"</camel.surefire.fork.additional-vmargs>
     </properties>
 
     <dependencies>
@@ -93,5 +95,29 @@
             <version>${project.version}</version>
             <scope>test</scope>
         </dependency>
+        <!-- deterministic simulation test of the consumer and its lock 
renewer (dst package) -->
+        <dependency>
+            <groupId>org.apache.camel</groupId>
+            <artifactId>camel-test-dst</artifactId>
+            <scope>test</scope>
+        </dependency>
     </dependencies>
+
+    <build>
+        <plugins>
+            <plugin>
+                <!-- sets the property net.bytebuddy:byte-buddy-agent:jar to 
the path of the agent of the DST tests -->
+                <groupId>org.apache.maven.plugins</groupId>
+                <artifactId>maven-dependency-plugin</artifactId>
+                <executions>
+                    <execution>
+                        <id>dst-agent-path</id>
+                        <goals>
+                            <goal>properties</goal>
+                        </goals>
+                    </execution>
+                </executions>
+            </plugin>
+        </plugins>
+    </build>
 </project>
diff --git 
a/components/camel-azure/camel-azure-servicebus/src/generated/resources/META-INF/org/apache/camel/component/azure/servicebus/azure-servicebus.json
 
b/components/camel-azure/camel-azure-servicebus/src/generated/resources/META-INF/org/apache/camel/component/azure/servicebus/azure-servicebus.json
index a79da4aee71f..1c868480c984 100644
--- 
a/components/camel-azure/camel-azure-servicebus/src/generated/resources/META-INF/org/apache/camel/component/azure/servicebus/azure-servicebus.json
+++ 
b/components/camel-azure/camel-azure-servicebus/src/generated/resources/META-INF/org/apache/camel/component/azure/servicebus/azure-servicebus.json
@@ -119,5 +119,8 @@
     "credentialType": { "index": 28, "kind": "parameter", "displayName": 
"Credential Type", "group": "security", "label": "security", "required": false, 
"type": "enum", "javaType": 
"org.apache.camel.component.azure.common.CredentialType", "enum": [ 
"AZURE_IDENTITY", "CONNECTION_STRING", "TOKEN_CREDENTIAL" ], "deprecated": 
false, "autowired": false, "secret": false, "defaultValue": 
"CONNECTION_STRING", "configurationClass": 
"org.apache.camel.component.azure.servicebus.ServiceBusConfigurat [...]
     "fullyQualifiedNamespace": { "index": 29, "kind": "parameter", 
"displayName": "Fully Qualified Namespace", "group": "security", "label": 
"security", "required": false, "type": "string", "javaType": 
"java.lang.String", "deprecated": false, "autowired": false, "secret": false, 
"configurationClass": 
"org.apache.camel.component.azure.servicebus.ServiceBusConfiguration", 
"configurationField": "configuration", "description": "Fully Qualified 
Namespace of the service bus" },
     "tokenCredential": { "index": 30, "kind": "parameter", "displayName": 
"Token Credential", "group": "security", "label": "security", "required": 
false, "type": "object", "javaType": 
"com.azure.core.credential.TokenCredential", "deprecated": false, "autowired": 
true, "secret": true, "security": "secret", "configurationClass": 
"org.apache.camel.component.azure.servicebus.ServiceBusConfiguration", 
"configurationField": "configuration", "description": "A TokenCredential for 
Azure AD authe [...]
+  },
+  "dstWorlds": {
+    "ServiceBusWorld": { "javaType": 
"org.apache.camel.component.azure.servicebus.dst.ServiceBusWorld", "fake": 
"FakeServiceBus: a queue in peek-lock mode (lock duration, delivery count, 
dead-letter queue) behind the processor and receiver clients of 
ServiceBusClientFactory", "covers": [ "the lock renewer of the consumer 
(PEEK_LOCK, maxAutoLockRenewDuration) with an asynchronous route", "processing 
that takes longer than the lock duration", "complete, abandon and dead-letter, 
also after  [...]
   }
 }
diff --git 
a/components/camel-azure/camel-azure-servicebus/src/test/java/org/apache/camel/component/azure/servicebus/dst/FakeServiceBus.java
 
b/components/camel-azure/camel-azure-servicebus/src/test/java/org/apache/camel/component/azure/servicebus/dst/FakeServiceBus.java
new file mode 100644
index 000000000000..e89ca458827c
--- /dev/null
+++ 
b/components/camel-azure/camel-azure-servicebus/src/test/java/org/apache/camel/component/azure/servicebus/dst/FakeServiceBus.java
@@ -0,0 +1,467 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.camel.component.azure.servicebus.dst;
+
+import java.time.Instant;
+import java.time.OffsetDateTime;
+import java.time.ZoneOffset;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.function.BiConsumer;
+import java.util.function.Consumer;
+
+import com.azure.core.amqp.exception.AmqpErrorCondition;
+import com.azure.core.amqp.exception.AmqpException;
+import com.azure.core.util.BinaryData;
+import com.azure.messaging.servicebus.ServiceBusErrorContext;
+import com.azure.messaging.servicebus.ServiceBusErrorSource;
+import com.azure.messaging.servicebus.ServiceBusException;
+import com.azure.messaging.servicebus.ServiceBusProcessorClient;
+import com.azure.messaging.servicebus.ServiceBusReceivedMessage;
+import com.azure.messaging.servicebus.ServiceBusReceivedMessageContext;
+import com.azure.messaging.servicebus.ServiceBusReceiverAsyncClient;
+import org.apache.camel.test.dst.Dst;
+import org.apache.camel.test.dst.DstExecutor;
+import org.apache.camel.test.dst.DstLedger;
+import org.apache.camel.test.dst.DstTask;
+import reactor.core.publisher.Mono;
+
+import static org.mockito.ArgumentMatchers.any;
+import static org.mockito.Mockito.doAnswer;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+
+/**
+ * A fake Azure Service Bus queue in peek-lock mode, and the clients 
camel-azure-servicebus gets from its
+ * ServiceBusClientFactory (Mockito mocks of the final SDK classes, whose 
behaviour is the queue below). Written from
+ * the Service Bus documentation, not from the Camel consumer:
+ * <ul>
+ * <li>[L] Peek-lock 
(https://learn.microsoft.com/azure/service-bus-messaging/message-transfers-locks-settlement):
 a
+ * received message is locked for the lock duration of the queue; "When the 
lock on a message is explicitly released or
+ * when the lock expires, the message goes back to the front of the retrieval 
order for redelivery"; "Whenever a message
+ * is delivered under a peek-lock, but is either explicitly abandoned or the 
lock has expired, the delivery count on the
+ * message is incremented. When the delivery count exceeds the limit, the 
message is moved to the DLQ"
+ * 
(https://learn.microsoft.com/azure/service-bus-messaging/service-bus-dead-letter-queues).</li>
+ * <li>[S] Settlement: complete removes the message, abandon releases the lock 
(the message is available again),
+ * deadLetter moves it to the dead-letter queue; "The Complete, DeadLetter, or 
RenewLock operations might fail due to
+ * network issues, if the held lock has expired" (assumed: the Java SDK 
reports it as a ServiceBusException with reason
+ * MESSAGE_LOCK_LOST). "Settling a message after you close the receiver throws 
an exception, and the message isn't
+ * settled" (dead-letter page): in azure-messaging-servicebus 7.17 the 
processor of a queue runs the V2 stack by
+ * default, where stop() closes the processor and its receiver, so a 
settlement after stop() or close() throws
+ * IllegalStateException. A settlement of a delivery that was settled already 
is a misuse (the SDK rejects it).</li>
+ * <li>[R] RenewMessageLock: "The client that owns the lock can extend it" 
(assumed: also the receiver client the
+ * consumer creates for its renewals, which is what the consumer relies on); 
the call takes a while, and the lock is
+ * extended by the lock duration from when the server handles it, which fails 
when the lock was lost by then. A renewal
+ * through a closed receiver client is a misuse.</li>
+ * <li>[P] The processor client (ServiceBusProcessorClient, auto-complete 
disabled by the factory): it receives messages
+ * and calls processMessage with up to maxConcurrentCalls calls at a time, and 
receives the next message for a call only
+ * when the call returned; stop and close end the receiving (and close the 
receiver, see [S]).</li>
+ * <li>[A] The lock renewal of the processor client while processMessage runs
+ * 
(https://learn.microsoft.com/java/api/com.azure.messaging.servicebus.servicebusclientbuilder.servicebusprocessorclientbuilder):
+ * "By default, a ServiceBusProcessorClient is configured with auto-completion 
and auto-lock renewal capabilities";
+ * maxAutoLockRenewDuration "Sets the amount of time to continue auto-renewing 
the lock. Setting Duration#ZERO or null
+ * disables auto-renewal"; "To keep the message locked, the client will have 
to continuously renew the message lock
+ * before its expiration. maxAutoLockRenewDuration controls how long the 
background renewal task runs" (the factory
+ * passes the maxAutoLockRenewDuration of the endpoint). In 
azure-messaging-servicebus 7.17 (javap of 7.17.16) the V2
+ * MessagePump begins the renewal of a message before it calls processMessage 
and disposes it when the call returns;
+ * LockRenewalOperation renews when the lock has min(half of the time left, 10 
seconds) left, until
+ * maxAutoLockRenewDuration has passed, and ends at the first failed renewal. 
The renewal goes through the receiver of
+ * the processor, so it ends when stop() or close() disposed it.</li>
+ * </ul>
+ * Not modelled: sessions, prefetch, deferral, scheduled messages, time to 
live, duplicate detection, topics and
+ * subscriptions (a queue only), faults and latency of the renewals of [A] 
(the faults are injected on the renewals of
+ * the consumer), connection loss and the processError callback.
+ * <p>
+ * Timing (one-stack rule): a receive takes the message at the end of its 
latency; settlement takes effect at once (it
+ * is made from completion callbacks); a renewal does not block (the lock 
renewer of the consumer and the renewal of [A]
+ * are background tasks), the latency of a renewal of the consumer is drawn 
when it is called and decides whether it
+ * comes in time.
+ */
+final class FakeServiceBus {
+
+    static final String QUEUE = "dst";
+    /** [A] LockRenewalOperation renews when the lock has min(half of the time 
left, this) left. */
+    static final long MAX_RENEWAL_BUFFER_MILLIS = 10_000;
+
+    enum State {
+        AVAILABLE,
+        LOCKED,
+        COMPLETED,
+        DEAD_LETTERED
+    }
+
+    static final class Msg {
+        final String id;
+        State state = State.AVAILABLE;
+        int deliveryCount;
+        String lockToken;
+        long lockedUntil;
+
+        Msg(String id) {
+            this.id = id;
+        }
+    }
+
+    private final Dst dst;
+    private final DstLedger ledger;
+    final Map<String, Msg> messages = new LinkedHashMap<>();
+    final List<String> violations = new ArrayList<>();
+    long lockDurationMillis = 30_000;
+    int maxDeliveryCount = 10;
+    double renewFailRate;
+    double settleFailRate;
+    /** The longest latency of a renewal while the faults are on (decided when 
the renewal is called). */
+    long renewLatencyMaxMillis;
+    boolean faults = true;
+    /** Called with the delivery ("id#n") when a message is delivered. */
+    Consumer<String> onDelivery = d -> {
+    };
+    /**
+     * Called with the delivery and the expiry of its lock (or -1 when it 
holds no lock) when an injected renewal fault
+     * hits it.
+     */
+    BiConsumer<String, Long> onRenewFault = (d, lockedUntil) -> {
+    };
+    /** Called with the delivery when a renewal (of the consumer or of the 
processor client) extended its lock. */
+    Consumer<String> onRenewed = d -> {
+    };
+    /** Called with "delivery STATE" when the consumer settles a delivery, 
before the outcome is known. */
+    Consumer<String> onSettle = d -> {
+    };
+    /**
+     * Called with "delivery outcome" after it: COMPLETED, ABANDONED, 
DEAD_LETTERED, fault, lockLost, closedByStop
+     * (after stop() and before close()) or closed (after close()).
+     */
+    Consumer<String> onSettled = d -> {
+    };
+    /** Called when a processor client is stopped or closed. */
+    Runnable onProcessorClosed = () -> {
+    };
+    /** Called with the delivery when its lock expires. */
+    Consumer<String> onLockExpired = d -> {
+    };
+    private final Set<String> settledDeliveries = new HashSet<>();
+    private int tokens;
+    private int receivers;
+
+    FakeServiceBus(Dst dst, DstLedger ledger) {
+        this.dst = dst;
+        this.ledger = ledger;
+    }
+
+    void send(String id) {
+        messages.put(id, new Msg(id));
+        ledger.record("sent", "q", id);
+    }
+
+    /** [L] locks that expired. */
+    void expire() {
+        for (Msg m : messages.values()) {
+            if (m.state == State.LOCKED && dst.now() >= m.lockedUntil) {
+                ledger.record("lockExpired", "q", m.id + "#" + 
m.deliveryCount);
+                onLockExpired.accept(m.id + "#" + m.deliveryCount);
+                available(m, "lock expired");
+            }
+        }
+    }
+
+    private void available(Msg m, String why) {
+        m.lockToken = null;
+        if (m.deliveryCount >= maxDeliveryCount) {
+            m.state = State.DEAD_LETTERED;
+            ledger.record("deadLettered", "q", m.id + " 
(MaxDeliveryCountExceeded)");
+        } else {
+            m.state = State.AVAILABLE;
+        }
+        dst.trace("fake: " + m.id + " " + why + " -> " + m.state);
+    }
+
+    boolean settled() {
+        expire();
+        return messages.values().stream()
+                .allMatch(m -> m.state == State.COMPLETED || m.state == 
State.DEAD_LETTERED);
+    }
+
+    String unsettled() {
+        return messages.values().stream().filter(m -> m.state != 
State.COMPLETED && m.state != State.DEAD_LETTERED)
+                .map(m -> m.id + "=" + m.state + "/d" + 
m.deliveryCount).toList().toString();
+    }
+
+    private static ServiceBusException lockLost(String token, 
ServiceBusErrorSource source) {
+        return new ServiceBusException(
+                new AmqpException(
+                        false, AmqpErrorCondition.MESSAGE_LOCK_LOST,
+                        "dst: the lock " + token + " is lost or expired", 
null),
+                source);
+    }
+
+    private static ServiceBusException transientFailure(ServiceBusErrorSource 
source) {
+        return new ServiceBusException(
+                new AmqpException(true, AmqpErrorCondition.SERVER_BUSY_ERROR, 
"dst: the server is busy", null), source);
+    }
+
+    /** The expiry of the lock with this token, or -1 when no message holds it 
(without expiring the locks first). */
+    private long lockedUntil(String token) {
+        for (Msg m : messages.values()) {
+            if (m.state == State.LOCKED && token.equals(m.lockToken)) {
+                return m.lockedUntil;
+            }
+        }
+        return -1;
+    }
+
+    private Msg locked(String token) {
+        expire();
+        for (Msg m : messages.values()) {
+            if (m.state == State.LOCKED && token.equals(m.lockToken)) {
+                return m;
+            }
+        }
+        return null;
+    }
+
+    /** ServiceBusClientFactory.createServiceBusProcessorClient: [P] a 
processor client on this queue. */
+    ServiceBusProcessorClient newProcessorClient(
+            int maxConcurrentCalls, long maxAutoLockRenewMillis,
+            Consumer<ServiceBusReceivedMessageContext> processMessage, 
Consumer<ServiceBusErrorContext> processError) {
+        ServiceBusProcessorClient client = 
mock(ServiceBusProcessorClient.class);
+        String name = "processor-" + receivers++;
+        DstExecutor calls = dst.executor(name + "-calls", maxConcurrentCalls);
+        DstExecutor receiver = dst.executor(name + "-receive", 1);
+        DstExecutor lockRenewal = dst.executor(name + "-lock-renewal", 
Integer.MAX_VALUE);
+        boolean[] running = new boolean[1];
+        // 0: open, 1: stopped (stop() closed the receiver), 2: closed
+        int[] closed = new int[1];
+        int[] inCalls = new int[1];
+        Runnable[] receive = new Runnable[1];
+        receive[0] = () -> {
+            if (!running[0]) {
+                return;
+            }
+            expire();
+            Msg next = inCalls[0] < maxConcurrentCalls
+                    ? messages.values().stream().filter(m -> m.state == 
State.AVAILABLE).findFirst().orElse(null)
+                    : null;
+            if (next == null) {
+                receiver.schedule(name + " receive", 100, receive[0]);
+                return;
+            }
+            // reads take effect at the end of their latency: the receive 
takes the message after it
+            dst.block(dst.between(5, 50), name + " receive");
+            expire();
+            if (!running[0] || next.state != State.AVAILABLE) {
+                receiver.schedule(name + " receive", 0, receive[0]);
+                return;
+            }
+            next.state = State.LOCKED;
+            next.deliveryCount++;
+            next.lockToken = "lock-" + tokens++;
+            next.lockedUntil = dst.now() + lockDurationMillis;
+            String delivery = next.id + "#" + next.deliveryCount;
+            ledger.record("delivered", "q", delivery);
+            onDelivery.accept(delivery);
+            ServiceBusReceivedMessageContext context = context(next, 
next.lockToken, delivery, closed);
+            inCalls[0]++;
+            String token = next.lockToken;
+            calls.schedule(name + " processMessage " + delivery, 0, () -> {
+                // [A] the renewal of the lock while the call runs, disposed 
when it returns
+                DstTask<?>[] renewal = new DstTask<?>[1];
+                if (maxAutoLockRenewMillis > 0) {
+                    Msg m = locked(token);
+                    if (m != null) {
+                        renewLockWhileProcessed(lockRenewal, renewal, token, 
delivery, m.lockedUntil,
+                                dst.now() + maxAutoLockRenewMillis, closed);
+                    }
+                }
+                try {
+                    processMessage.accept(context);
+                } finally {
+                    if (renewal[0] != null) {
+                        renewal[0].cancel(false);
+                    }
+                    inCalls[0]--;
+                }
+            });
+            receiver.schedule(name + " receive", 0, receive[0]);
+        };
+        doAnswer(i -> {
+            if (closed[0] != 0) {
+                violations.add(name + " started after it was closed");
+                return null;
+            }
+            if (!running[0]) {
+                running[0] = true;
+                dst.trace("fake: " + name + " start");
+                receiver.schedule(name + " receive", 0, receive[0]);
+            }
+            return null;
+        }).when(client).start();
+        doAnswer(i -> {
+            // [P] [S] V2 stack: stop() closes the processor and its receiver
+            running[0] = false;
+            closed[0] = Math.max(closed[0], 1);
+            dst.trace("fake: " + name + " stop (closes the receiver)");
+            onProcessorClosed.run();
+            return null;
+        }).when(client).stop();
+        doAnswer(i -> {
+            running[0] = false;
+            closed[0] = 2;
+            dst.trace("fake: " + name + " close");
+            onProcessorClosed.run();
+            return null;
+        }).when(client).close();
+        when(client.isRunning()).thenAnswer(i -> running[0]);
+        return client;
+    }
+
+    /**
+     * [A] The next renewal of the processor client for a message whose 
processMessage call runs: when the lock it knows
+     * of has min(half of the time left, 10 seconds) left; it ends once 
maxAutoLockRenewDuration has passed, when the
+     * receiver of the processor is closed, or at the first failed renewal 
(the lock was lost).
+     */
+    private void renewLockWhileProcessed(
+            DstExecutor lockRenewal, DstTask<?>[] renewal, String token, 
String delivery, long lockedUntil, long until,
+            int[] closed) {
+        long left = lockedUntil - dst.now();
+        long delay = left < 400 ? 0 : left - Math.min(left / 2, 
MAX_RENEWAL_BUFFER_MILLIS);
+        renewal[0] = lockRenewal.schedule("processor lock renewal of " + 
delivery, delay, () -> {
+            renewal[0] = null;
+            if (closed[0] != 0 || dst.now() >= until) {
+                return;
+            }
+            Msg m = locked(token);
+            if (m == null) {
+                ledger.record("processorRenewLockLost", "q", delivery);
+                return;
+            }
+            m.lockedUntil = dst.now() + lockDurationMillis;
+            ledger.record("processorRenewed", "q", delivery);
+            onRenewed.accept(delivery);
+            renewLockWhileProcessed(lockRenewal, renewal, token, delivery, 
m.lockedUntil, until, closed);
+        });
+    }
+
+    private ServiceBusReceivedMessageContext context(Msg m, String token, 
String delivery, int[] closed) {
+        ServiceBusReceivedMessage message = 
mock(ServiceBusReceivedMessage.class);
+        when(message.getBody()).thenReturn(BinaryData.fromString(m.id));
+        when(message.getMessageId()).thenReturn(m.id);
+        when(message.getLockToken()).thenReturn(token);
+        when(message.getDeliveryCount()).thenReturn((long) m.deliveryCount);
+        when(message.getLockedUntil())
+                
.thenReturn(OffsetDateTime.ofInstant(Instant.ofEpochMilli(Dst.EPOCH_MILLIS + 
m.lockedUntil), ZoneOffset.UTC));
+        when(message.getApplicationProperties()).thenReturn(Map.of());
+        ServiceBusReceivedMessageContext context = 
mock(ServiceBusReceivedMessageContext.class);
+        when(context.getMessage()).thenReturn(message);
+        doAnswer(i -> {
+            settle(token, delivery, State.COMPLETED, 
ServiceBusErrorSource.COMPLETE, closed[0]);
+            return null;
+        }).when(context).complete();
+        doAnswer(i -> {
+            settle(token, delivery, State.AVAILABLE, 
ServiceBusErrorSource.ABANDON, closed[0]);
+            return null;
+        }).when(context).abandon();
+        doAnswer(i -> {
+            settle(token, delivery, State.DEAD_LETTERED, 
ServiceBusErrorSource.UNKNOWN, closed[0]);
+            return null;
+        }).when(context).deadLetter();
+        doAnswer(i -> {
+            settle(token, delivery, State.DEAD_LETTERED, 
ServiceBusErrorSource.UNKNOWN, closed[0]);
+            return null;
+        }).when(context).deadLetter(any());
+        return context;
+    }
+
+    /** [S] a settlement with the lock token of a delivery. */
+    private void settle(String token, String delivery, State to, 
ServiceBusErrorSource source, int closed) {
+        onSettle.accept(delivery + " " + to);
+        // the SDK checks a disposed receiver first, and marks a message 
settled only once a disposition succeeded
+        if (closed != 0) {
+            ledger.record("settleAfterClose", "q", delivery + " " + to);
+            onSettled.accept(delivery + (closed == 1 ? " closedByStop" : " 
closed"));
+            throw new IllegalStateException("Cannot perform operation on a 
disposed receiver.");
+        }
+        if (settledDeliveries.contains(delivery)) {
+            violations.add(delivery + " settled again (" + to + ")");
+            throw new IllegalArgumentException("The message has either been 
deleted or already settled.");
+        }
+        if (faults && dst.chance(settleFailRate)) {
+            ledger.record("settleFault", "q", delivery);
+            onSettled.accept(delivery + " fault");
+            throw Dst.injected(transientFailure(source));
+        }
+        Msg m = locked(token);
+        if (m == null) {
+            ledger.record("lockLost", "q", delivery + " " + to);
+            onSettled.accept(delivery + " lockLost");
+            throw lockLost(token, source);
+        }
+        ledger.record(to.name().toLowerCase(), "q", delivery);
+        settledDeliveries.add(delivery);
+        onSettled.accept(delivery + " " + (to == State.AVAILABLE ? "ABANDONED" 
: to.name()));
+        if (to == State.AVAILABLE) {
+            available(m, "abandoned");
+        } else {
+            m.state = to;
+            m.lockToken = null;
+        }
+    }
+
+    /** ServiceBusClientFactory.createServiceBusReceiverAsyncClient: [R] the 
lock renewals of the consumer. */
+    ServiceBusReceiverAsyncClient newRenewalClient() {
+        ServiceBusReceiverAsyncClient client = 
mock(ServiceBusReceiverAsyncClient.class);
+        boolean[] closed = new boolean[1];
+        doAnswer(i -> {
+            closed[0] = true;
+            return null;
+        }).when(client).close();
+        
when(client.renewMessageLock(any(ServiceBusReceivedMessage.class))).thenAnswer(i
 -> {
+            ServiceBusReceivedMessage message = i.getArgument(0);
+            String token = message.getLockToken();
+            String delivery = message.getMessageId() + "#" + 
message.getDeliveryCount();
+            if (closed[0]) {
+                violations.add("a renewal of " + delivery + " through a closed 
receiver client");
+                return Mono.error(new IllegalStateException("Cannot perform 
operation on a disposed receiver."));
+            }
+            if (settledDeliveries.contains(delivery)) {
+                violations.add("a renewal of " + delivery + " after it was 
settled");
+            }
+            if (faults && dst.chance(renewFailRate)) {
+                ledger.record("renewFault", "q", delivery);
+                onRenewFault.accept(delivery, lockedUntil(token));
+                return 
Mono.error(Dst.injected(transientFailure(ServiceBusErrorSource.RENEW_LOCK)));
+            }
+            // the server handles the renewal after its latency: a lock that 
expires before that is lost
+            long latency = faults && renewLatencyMaxMillis > 0 ? 
dst.between(1, renewLatencyMaxMillis) : 1;
+            Msg m = locked(token);
+            if (m == null || dst.now() + latency >= m.lockedUntil) {
+                ledger.record("renewLockLost", "q", delivery);
+                return Mono.error(lockLost(token, 
ServiceBusErrorSource.RENEW_LOCK));
+            }
+            m.lockedUntil = dst.now() + latency + lockDurationMillis;
+            ledger.record("renewed", "q", delivery);
+            onRenewed.accept(delivery);
+            return Mono.just(
+                    
OffsetDateTime.ofInstant(Instant.ofEpochMilli(Dst.EPOCH_MILLIS + 
m.lockedUntil), ZoneOffset.UTC));
+        });
+        return client;
+    }
+}
diff --git 
a/components/camel-azure/camel-azure-servicebus/src/test/java/org/apache/camel/component/azure/servicebus/dst/ServiceBusConsumerDstTest.java
 
b/components/camel-azure/camel-azure-servicebus/src/test/java/org/apache/camel/component/azure/servicebus/dst/ServiceBusConsumerDstTest.java
new file mode 100644
index 000000000000..87db721adf59
--- /dev/null
+++ 
b/components/camel-azure/camel-azure-servicebus/src/test/java/org/apache/camel/component/azure/servicebus/dst/ServiceBusConsumerDstTest.java
@@ -0,0 +1,37 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.camel.component.azure.servicebus.dst;
+
+import org.apache.camel.test.dst.Dst;
+import org.apache.camel.test.dst.DstInstrument;
+import org.apache.camel.test.dst.DstTest;
+
+/**
+ * Deterministic simulation test of the camel-azure-servicebus consumer in 
peek-lock mode against a fake queue, with an
+ * asynchronous route slower than the lock duration, failed lock renewals and 
settlements, and route stops while
+ * messages are processed: no message is delivered again while the consumer 
holds it, a message is settled as its
+ * exchange ended, and once the faults cleared every message is completed or 
dead-lettered. A failing seed is replayed
+ * with {@code -Ddst.seed=<seed>}; {@code -Ddst.seeds=5000} runs more seeds.
+ */
+@DstInstrument("org.apache.camel.component.azure.servicebus")
+class ServiceBusConsumerDstTest {
+
+    @DstTest(seeds = 1000)
+    void noMessageIsDeliveredAgainWhileTheConsumerHoldsIt(Dst dst) throws 
Exception {
+        new ServiceBusWorld(dst).run();
+    }
+}
diff --git 
a/components/camel-azure/camel-azure-servicebus/src/test/java/org/apache/camel/component/azure/servicebus/dst/ServiceBusWorld.java
 
b/components/camel-azure/camel-azure-servicebus/src/test/java/org/apache/camel/component/azure/servicebus/dst/ServiceBusWorld.java
new file mode 100644
index 000000000000..543e3fb45cf4
--- /dev/null
+++ 
b/components/camel-azure/camel-azure-servicebus/src/test/java/org/apache/camel/component/azure/servicebus/dst/ServiceBusWorld.java
@@ -0,0 +1,582 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.camel.component.azure.servicebus.dst;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.function.Consumer;
+
+import com.azure.messaging.servicebus.ServiceBusErrorContext;
+import com.azure.messaging.servicebus.ServiceBusReceivedMessageContext;
+import org.apache.camel.AsyncCallback;
+import org.apache.camel.Exchange;
+import org.apache.camel.Route;
+import org.apache.camel.ServiceStatus;
+import org.apache.camel.builder.RouteBuilder;
+import org.apache.camel.component.azure.servicebus.ServiceBusComponent;
+import org.apache.camel.component.azure.servicebus.ServiceBusConfiguration;
+import org.apache.camel.component.azure.servicebus.ServiceBusConstants;
+import org.apache.camel.component.azure.servicebus.ServiceBusEndpoint;
+import 
org.apache.camel.component.azure.servicebus.client.ServiceBusClientFactory;
+import org.apache.camel.impl.DefaultCamelContext;
+import org.apache.camel.spi.ShutdownAware;
+import org.apache.camel.support.AsyncProcessorSupport;
+import org.apache.camel.support.service.ServiceHelper;
+import org.apache.camel.test.dst.Dst;
+import org.apache.camel.test.dst.DstCamel;
+import org.apache.camel.test.dst.DstExecutor;
+import org.apache.camel.test.dst.DstLedger;
+import org.apache.camel.test.dst.DstWorld;
+import org.apache.camel.test.dst.Invariants;
+
+import static org.mockito.ArgumentMatchers.any;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+
+/**
+ * The world around a camel-azure-servicebus consumer in peek-lock mode: a 
fake queue ({@link FakeServiceBus}), a
+ * sender, a route with an asynchronous step (a downstream call) that is 
sometimes longer than the lock duration and
+ * sometimes fails, and an operator that stops and starts the route while 
messages are processed. Faults: lock renewals
+ * fail or take long, settlements fail.
+ * <p>
+ * The route is asynchronous, so processMessage returns at once and the lock 
of a routed message is kept only by the
+ * lock renewer of the consumer (maxAutoLockRenewDuration), which renews every 
message whose lock expires before the run
+ * after the next one of its period task (every 10 seconds, or a second 
later). A message the consumer holds in
+ * processMessage without routing it keeps its lock through the renewal of the 
processor client while the call runs (see
+ * FakeServiceBus [A]); the invariants count only the routed deliveries as 
held by the consumer. The seed picks the lock
+ * duration of the queue, the max delivery count, maxConcurrentCalls (with the 
asynchronous route it bounds only the
+ * concurrent processMessage calls), maxAutoLockRenewDuration (60 or 300 
seconds), enableDeadLettering, the latency of
+ * the renewals and the fault rates.
+ * <p>
+ * Properties: no lock is lost, and no message is delivered again, while the 
consumer holds the message, unless an
+ * injected renewal of its lock failed less than one run of the lock renewer 
before the lock expired, the route was
+ * stopped while it was processed (once a stop timed out the consumer stops 
renewing, and the exchange goes on), or
+ * maxAutoLockRenewDuration passed. The consumer renews a lock again after a 
transient renewal error (CAMEL-26969), but
+ * only at its next run (CAMEL-27066): the lock renewer runs every 10 seconds 
or a second later, so a transient renewal
+ * error in the last 11 seconds of a lock (plus the latency of the renewal) 
can still lose it. An earlier failed renewal
+ * is no excuse, so a renewer that gave up after one error would fail. {@code 
-Ddst.servicebus.strictRenew=true} does
+ * not excuse a failed renewal at all, and still fails on some seeds until 
CAMEL-27066 is fixed. The message of a
+ * completed exchange is completed, unless its settlement failed, found its 
lock lost under one of those excuses, or
+ * came after the close() of a stop that timed out while the exchange was in 
flight (by design); a graceful stop keeps
+ * the processor client open for the exchanges in flight (the V2 processor of 
the SDK closes its receiver on stop(), see
+ * FakeServiceBus; CAMEL-26970), so a settlement inside the graceful wait must 
find it open; a message is completed only
+ * after its exchange completed, and abandoned or dead-lettered only after it 
failed (or was in flight when a stop timed
+ * out, see below); once the faults cleared every message is completed or 
dead-lettered; the consumer settles each
+ * delivery at most once and renews only what it holds, through an open client.
+ * <p>
+ * The operator stops the route as DefaultShutdownStrategy does (its 
ShutdownTask), with the wait of the strategy as
+ * tasks of the operator (see {@link #gracefulStop}): a processMessage call 
that waits until the consumer stops would
+ * otherwise sit above the stop on the one stack, and could not see the stop. 
When the stop times out, the consumer is
+ * forced to stop as the strategy does, and the exchanges still in flight fail 
(their route rejects them).
+ * <p>
+ * Not explored: sessions, receiveAndDelete mode, a processorClient given by 
the user (the consumer does not renew its
+ * locks), a synchronous route (a processMessage call that lasts as long as 
the exchange), maxAutoLockRenewDuration=0
+ * (no renewal by the consumer or by the processor client), topics.
+ */
+@DstWorld(component = "azure-servicebus",
+          fake = "FakeServiceBus: a queue in peek-lock mode (lock duration, 
delivery count, dead-letter queue) behind"
+                 + " the processor and receiver clients of 
ServiceBusClientFactory",
+          covers = {
+                  "the lock renewer of the consumer (PEEK_LOCK, 
maxAutoLockRenewDuration) with an asynchronous route",
+                  "processing that takes longer than the lock duration",
+                  "complete, abandon and dead-letter, also after the lock was 
lost",
+                  "maxConcurrentCalls of one to three (with the asynchronous 
route it bounds the processMessage calls)",
+                  "route stop and start while messages are processed" },
+          faults = {
+                  "a lock renewal fails", "a lock renewal takes long", "a 
settlement fails", "processing fails",
+                  "processing is slower than the lock duration" },
+          invariants = {
+                  "no lock is lost and no message is delivered again while the 
consumer holds it, unless a renewal"
+                         + " failed less than one renewer run before the lock 
expired, the route was stopped or"
+                         + " maxAutoLockRenewDuration passed",
+                  "the message of a completed exchange is completed, unless 
its settlement failed, its lock was lost"
+                                                               + " as excused 
above, or a forced stop closed the client first",
+                  "a message is completed only after its exchange completed",
+                  "a message is abandoned or dead-lettered only after its 
exchange failed",
+                  "every message is completed or dead-lettered once the faults 
cleared",
+                  "the consumer settles a delivery at most once and renews 
only what it holds, through an open client" },
+          issues = { "CAMEL-25701", "CAMEL-26956", "CAMEL-26969", 
"CAMEL-26970", "CAMEL-27066" })
+final class ServiceBusWorld {
+
+    static final String ROUTE = "servicebus";
+    /**
+     * Excuses a lost lock only when the renewal found it lost, not after an 
injected transient renewal failure: the
+     * consumer retries a failed renewal only at its next run (CAMEL-27066), 
so this fails on some seeds until it is
+     * fixed.
+     */
+    static final boolean STRICT_RENEW = 
Boolean.getBoolean("dst.servicebus.strictRenew");
+    /** The longest time between two runs of the lock renewer of the consumer: 
every 10 seconds, or a second later. */
+    static final long RENEWER_RUN_MILLIS = 11_000;
+
+    private final Dst dst;
+    private final DstLedger ledger;
+    private final FakeServiceBus bus;
+    private final int maxConcurrentCalls;
+    private final boolean deadLettering;
+    private final double processFailRate;
+    private final long maxAutoLockRenewDuration;
+    private final List<String> violations = new ArrayList<>();
+    private final List<String> lostWhileHeld = new ArrayList<>();
+    /** The time each delivery was routed (the lock renewer of the consumer 
counts maxAutoLockRenewDuration from it). */
+    private final Map<String, Long> routedAt = new HashMap<>();
+    /**
+     * The outcome of the settlement of each delivery (COMPLETED, ABANDONED, 
DEAD_LETTERED, fault, lockLost,
+     * closedByStop, closed).
+     */
+    private final Map<String, String> outcomes = new HashMap<>();
+    private final List<String> notCompleted = new ArrayList<>();
+    private final List<String> redeliveredWhileHeld = new ArrayList<>();
+    private final List<String> wrongSettlements = new ArrayList<>();
+    /**
+     * The deliveries ("id#n") whose exchange is in progress, by message id: 
routed and not done yet. A delivery the
+     * consumer holds in processMessage without routing it is not counted (the 
processor client renews its lock).
+     */
+    private final Map<String, String> held = new HashMap<>();
+    /**
+     * Every delivery whose exchange is in progress (a message delivered again 
while an earlier delivery of it is still
+     * in progress has two).
+     */
+    private final Set<String> inProgress = new HashSet<>();
+    /**
+     * The deliveries whose last renewal failed less than one run of the lock 
renewer (RENEWER_RUN_MILLIS, plus the
+     * latency of a renewal) before their lock expired: the retry at the next 
run comes too late (CAMEL-27066). A later
+     * renewal that extended the lock removes the delivery again.
+     */
+    private final Set<String> renewFaulted = new HashSet<>();
+    /** The deliveries in progress when the route was stopped: the consumer 
stops renewing their locks. */
+    private final Set<String> heldAtStop = new HashSet<>();
+    /**
+     * The deliveries in progress when a graceful stop timed out and the 
consumer was forced to stop: stopping the
+     * consumer stops the processor of its route, which then rejects the 
exchange (it fails, and its message is
+     * abandoned or dead-lettered, through the closed client), even when the 
downstream call answered.
+     */
+    private final Set<String> forcedAtStop = new HashSet<>();
+    /** The deliveries of forcedAtStop abandoned or dead-lettered: that 
settlement must find the client closed. */
+    private final Set<String> forcedSettlements = new HashSet<>();
+    /** The deliveries routed, or settled (an abandon when routing failed), by 
the consumer. */
+    private final Set<String> routedOrSettled = new HashSet<>();
+    /**
+     * Whether the operator stopped the route and has not started it again 
(from deferShutdown until the restart): with
+     * the CAMEL-26970 fix a call that comes after the stop closed the client 
returns without routing, as the consumer
+     * stays in its stopping state until it is started again.
+     */
+    private boolean stopping;
+    private final Set<String> completed = new HashSet<>();
+    private final Set<String> failed = new HashSet<>();
+    private boolean faults = true;
+    /** Exchanges whose asynchronous step has not answered yet. */
+    private int inFlight;
+    private int sendsLeft;
+    private int opsLeft;
+    private boolean finished;
+    private String finalStatus;
+    private boolean finalSettled;
+
+    ServiceBusWorld(Dst dst) {
+        this.dst = dst;
+        this.ledger = new DstLedger(dst);
+        this.bus = new FakeServiceBus(dst, ledger);
+        bus.lockDurationMillis = dst.pick(List.of(30_000L, 60_000L));
+        bus.maxDeliveryCount = dst.pick(List.of(3, 10));
+        bus.renewFailRate = dst.pick(List.of(0.0, 0.05, 0.2));
+        bus.settleFailRate = dst.pick(List.of(0.0, 0.05));
+        bus.renewLatencyMaxMillis = dst.pick(List.of(0L, 2000L));
+        this.maxAutoLockRenewDuration = dst.pick(List.of(60_000L, 300_000L));
+        this.maxConcurrentCalls = dst.pick(List.of(1, 2, 3));
+        this.deadLettering = dst.chance(0.3);
+        this.processFailRate = dst.pick(List.of(0.0, 0.05, 0.2));
+        bus.onDelivery = this::delivered;
+        bus.onRenewFault = this::renewFault;
+        bus.onRenewed = renewFaulted::remove;
+        bus.onSettle = this::settling;
+        bus.onSettled = this::settled;
+        bus.onLockExpired = this::lockExpired;
+        // what is in flight when the processor client stops: the stop happens 
inside the route stop, after the
+        // operator took its own snapshot, and a message can be delivered in 
between
+        bus.onProcessorClosed = () -> heldAtStop.addAll(inProgress);
+        dst.trace("world: " + config());
+    }
+
+    private String config() {
+        return "lockDuration=" + bus.lockDurationMillis + " maxDeliveryCount=" 
+ bus.maxDeliveryCount
+               + " maxConcurrentCalls=" + maxConcurrentCalls + " 
maxAutoLockRenewDuration=" + maxAutoLockRenewDuration
+               + " enableDeadLettering=" + deadLettering + " renewFail=" + 
bus.renewFailRate + " renewLatencyMax="
+               + bus.renewLatencyMaxMillis + " settleFail=" + 
bus.settleFailRate + " processFail=" + processFailRate;
+    }
+
+    @SuppressWarnings("unchecked")
+    void run() throws Exception {
+        DefaultCamelContext context = DstCamel.newContext(dst);
+        ServiceBusClientFactory factory = mock(ServiceBusClientFactory.class);
+        
when(factory.createServiceBusProcessorClient(any(ServiceBusConfiguration.class),
 any(Consumer.class),
+                any(Consumer.class))).thenAnswer(
+                        i -> bus.newProcessorClient(
+                                i.<ServiceBusConfiguration> 
getArgument(0).getMaxConcurrentCalls(),
+                                i.<ServiceBusConfiguration> 
getArgument(0).getMaxAutoLockRenewDuration(),
+                                
checkedProcessMessage(i.<Consumer<ServiceBusReceivedMessageContext>> 
getArgument(1)),
+                                i.<Consumer<ServiceBusErrorContext>> 
getArgument(2)));
+        
when(factory.createServiceBusReceiverAsyncClient(any(ServiceBusConfiguration.class)))
+                .thenAnswer(i -> bus.newRenewalClient());
+        context.addComponent("azure-servicebus", new ServiceBusComponent());
+        String uri = "azure-servicebus:" + FakeServiceBus.QUEUE
+                     + 
"?connectionString=RAW(Endpoint=sb://dst.servicebus.windows.net/;SharedAccessKeyName=dst;"
+                     + 
"SharedAccessKey=dst)&serviceBusReceiveMode=PEEK_LOCK&maxConcurrentCalls=" + 
maxConcurrentCalls
+                     + "&enableDeadLettering=" + deadLettering + 
"&maxAutoLockRenewDuration=" + maxAutoLockRenewDuration;
+        DstExecutor downstream = dst.executor("downstream", Integer.MAX_VALUE);
+        context.addRoutes(new RouteBuilder() {
+            @Override
+            public void configure() {
+                // the injected processing failures are expected: no error 
handler log for each of them
+                errorHandler(defaultErrorHandler().logExhausted(false));
+                from(uri).routeId(ROUTE).autoStartup(false)
+                        .process(new AsyncProcessorSupport() {
+                            @Override
+                            public boolean process(Exchange exchange, 
AsyncCallback callback) {
+                                return downstream(exchange, callback, 
downstream);
+                            }
+                        });
+            }
+        });
+        ((ServiceBusEndpoint) 
context.getEndpoint(uri)).setServiceBusClientFactory(factory);
+        context.start();
+        DstExecutor operator = dst.executor("operator", 1);
+        try {
+            operator.schedule("start route", 0, () -> startRoute(context));
+            planSends();
+            planOperations(context, operator);
+            operator.schedule("finisher", 1000, () -> finisher(context, 
operator, -1));
+            dst.runUntil(() -> finished, "the finisher");
+            dst.check(
+                    Invariants.of("the consumer settles a delivery at most 
once and renews only what it holds, through an"
+                                  + " open client",
+                            () -> bus.violations.isEmpty() && 
violations.isEmpty()
+                                    ? null : bus.violations + " " + violations 
+ " (" + config() + ")"),
+                    Invariants.of("no lock is lost and no message is delivered 
again while the consumer holds it,"
+                                  + " unless a renewal failed less than one 
renewer run before the lock expired, the"
+                                  + " route was stopped or 
maxAutoLockRenewDuration passed",
+                            () -> redeliveredWhileHeld.isEmpty() && 
lostWhileHeld.isEmpty()
+                                    ? null : redeliveredWhileHeld + " " + 
lostWhileHeld + " (" + config() + ")"),
+                    Invariants.of("the message of a completed exchange is 
completed, unless its settlement failed, its"
+                                  + " lock was lost as excused above, or a 
forced stop closed the client first",
+                            () -> notCompleted.isEmpty() ? null : notCompleted 
+ " (" + config() + ")"),
+                    Invariants.of("a message is completed only after its 
exchange completed",
+                            () -> wrongSettlements.stream().noneMatch(s -> 
s.contains("COMPLETED"))
+                                    ? null : wrongSettlements + " (" + 
config() + ")"),
+                    Invariants.of("a message is abandoned or dead-lettered 
only after its exchange failed",
+                            () -> wrongSettlements.stream().allMatch(s -> 
s.contains("COMPLETED"))
+                                    ? null : wrongSettlements + " (" + 
config() + ")"),
+                    Invariants.of("every message is completed or dead-lettered 
once the faults cleared",
+                            () -> finalSettled && "Started".equals(finalStatus)
+                                    ? null
+                                    : "route " + finalStatus + ", unsettled " 
+ bus.unsettled() + " (" + config() + ")"));
+        } finally {
+            dst.dropPending();
+            // with an exchange still in flight (its answer was dropped) the 
shutdown would wait for it for ever on the
+            // one stack
+            if (inFlight == 0) {
+                context.stop();
+            }
+        }
+    }
+
+    /**
+     * The processMessage callback of the consumer, checked: unless the 
operator stopped the route and has not started
+     * it again, a delivered message is routed or settled by the time the call 
returns (the route is synchronous up to
+     * its downstream call). Once the route is stopped the consumer may hold 
the message in the call, unrouted, until it
+     * stops, and return without routing it until the route is started again.
+     */
+    private Consumer<ServiceBusReceivedMessageContext> checkedProcessMessage(
+            Consumer<ServiceBusReceivedMessageContext> processMessage) {
+        return context -> {
+            boolean duringStop = stopping;
+            String delivery = context.getMessage().getMessageId() + "#" + 
context.getMessage().getDeliveryCount();
+            processMessage.accept(context);
+            if (!duringStop && !routedOrSettled.contains(delivery)) {
+                violations.add(delivery + " neither routed nor settled when 
processMessage returned at t=" + dst.now());
+            }
+        };
+    }
+
+    /** The asynchronous step of the route: a downstream call that answers 
after a latency, and may fail. */
+    private boolean downstream(Exchange exchange, AsyncCallback callback, 
DstExecutor downstream) {
+        String id = 
exchange.getMessage().getHeader(ServiceBusConstants.MESSAGE_ID, String.class);
+        Long count = 
exchange.getMessage().getHeader(ServiceBusConstants.DELIVERY_COUNT, Long.class);
+        String delivery = id + "#" + count;
+        ledger.record("routed", "q", delivery);
+        // from now on the consumer holds the message, until its exchange is 
done
+        held.put(id, delivery);
+        inProgress.add(delivery);
+        routedOrSettled.add(delivery);
+        routedAt.put(delivery, dst.now());
+        long latency = faults && dst.chance(0.2)
+                ? dst.between(bus.lockDurationMillis / 2, 3 * 
bus.lockDurationMillis)
+                : dst.between(10, 500);
+        boolean fail = faults && dst.chance(processFailRate);
+        inFlight++;
+        downstream.schedule("answer for " + delivery, latency, () -> {
+            inFlight--;
+            if (fail) {
+                failed.add(delivery);
+                ledger.record("failed", "q", delivery);
+                exchange.setException(Dst.injected(new 
IllegalStateException("dst: processing failed for " + delivery)));
+            } else {
+                completed.add(delivery);
+                ledger.record("completed", "q", delivery);
+            }
+            held.remove(id, delivery);
+            inProgress.remove(delivery);
+            callback.done(false);
+        });
+        return false;
+    }
+
+    /** The queue delivered a message: it must not be one whose exchange is in 
progress. */
+    private void delivered(String delivery) {
+        String id = delivery.substring(0, delivery.indexOf('#'));
+        String previous = held.get(id);
+        if (previous != null && !excused(previous)) {
+            redeliveredWhileHeld.add(delivery + " at t=" + dst.now() + " while 
" + previous + " is processed");
+        }
+    }
+
+    /**
+     * An injected renewal failure excuses the loss of the lock only when the 
next run of the lock renewer comes too
+     * late to renew it: the renewer runs every 10 seconds or a second later 
(the period task scheduler ticks every
+     * second), and the renewal itself takes up to renewLatencyMaxMillis.
+     */
+    private void renewFault(String delivery, long lockedUntil) {
+        if (lockedUntil < 0) {
+            // the lock was lost already: its loss was judged when it expired
+            return;
+        }
+        if (lockedUntil - dst.now() <= RENEWER_RUN_MILLIS + 
bus.renewLatencyMaxMillis) {
+            renewFaulted.add(delivery);
+        } else {
+            renewFaulted.remove(delivery);
+        }
+    }
+
+    /**
+     * Why the consumer may lose the lock of a delivery it holds: an injected 
renewal failure in the last run of the
+     * lock renewer before the lock expired (the consumer renews again only at 
its next run, CAMEL-27066), a route stop
+     * while it was processed (the consumer stops renewing once the stop timed 
out), or a hold longer than
+     * maxAutoLockRenewDuration.
+     */
+    private boolean excused(String delivery) {
+        Long at = routedAt.get(delivery);
+        return !STRICT_RENEW && renewFaulted.contains(delivery) || 
heldAtStop.contains(delivery)
+                || at != null && dst.now() >= at + maxAutoLockRenewDuration;
+    }
+
+    /** The queue let the lock of a delivery expire: the consumer must renew 
the locks of what it holds. */
+    private void lockExpired(String delivery) {
+        String id = delivery.substring(0, delivery.indexOf('#'));
+        if (delivery.equals(held.get(id)) && !excused(delivery)) {
+            lostWhileHeld.add("the lock of " + delivery + " expired at t=" + 
dst.now() + " while it is processed");
+        }
+    }
+
+    private void settled(String outcome) {
+        int space = outcome.indexOf(' ');
+        String delivery = outcome.substring(0, space);
+        String result = outcome.substring(space + 1);
+        if (forcedSettlements.remove(delivery) && !"closed".equals(result)) {
+            violations.add(delivery + " in flight at a forced stop was settled 
with outcome " + result
+                           + " (only through the closed client is expected)");
+        }
+        // whether a lost lock is excused is decided when the settlement finds 
it lost
+        outcomes.put(delivery, "lockLost".equals(result) && excused(delivery) 
? "lockLost (excused)" : result);
+    }
+
+    /** A completed exchange must complete its message, unless the settlement 
failed for an excused reason. */
+    private void checkCompletions() {
+        for (String delivery : completed.stream().sorted().toList()) {
+            String outcome = outcomes.get(delivery);
+            boolean ok = "COMPLETED".equals(outcome) || "fault".equals(outcome)
+                    || "lockLost (excused)".equals(outcome)
+                    // a settlement after the consumer's own close() is one of 
an exchange that was in flight when the
+                    // stop timed out (by design); a graceful stop keeps the 
client open, so "closedByStop" is not excused
+                    || "closed".equals(outcome);
+            if (!ok) {
+                notCompleted.add(delivery + " completed but its settlement was 
" + outcome);
+            }
+        }
+    }
+
+    private void settling(String settlement) {
+        String delivery = settlement.substring(0, settlement.indexOf(' '));
+        boolean complete = settlement.endsWith("COMPLETED");
+        routedOrSettled.add(delivery);
+        if (!complete && !failed.contains(delivery) && 
forcedAtStop.contains(delivery)) {
+            forcedSettlements.add(delivery);
+        } else if (complete && !completed.contains(delivery) || !complete && 
!failed.contains(delivery)) {
+            wrongSettlements.add(settlement + " at t=" + dst.now());
+        }
+    }
+
+    private void finisher(DefaultCamelContext context, DstExecutor operator, 
long deadline) {
+        if (deadline == -1 && (sendsLeft > 0 || opsLeft > 0)) {
+            operator.schedule("finisher", 500, () -> finisher(context, 
operator, -1));
+            return;
+        }
+        if (deadline == -1) {
+            long slow = dst.between(0, 10_000);
+            dst.trace("world: operations done, faults for another " + slow + " 
ms");
+            operator.schedule("finisher", slow, () -> finisher(context, 
operator, -2));
+            return;
+        }
+        if (deadline == -2) {
+            faults = false;
+            bus.faults = false;
+            dst.trace("world: faults cleared");
+            long d = dst.now() + 60 * 60_000L;
+            operator.schedule("finisher", 500, () -> finisher(context, 
operator, d));
+            return;
+        }
+        ServiceStatus status = 
context.getRouteController().getRouteStatus(ROUTE);
+        boolean settled = bus.settled();
+        if ((!settled || inFlight > 0 || status != ServiceStatus.Started) && 
dst.now() < deadline) {
+            operator.schedule("finisher", 500, () -> finisher(context, 
operator, deadline));
+            return;
+        }
+        finalStatus = String.valueOf(status);
+        finalSettled = settled;
+        if (inFlight > 0) {
+            violations.add(inFlight + " exchanges still in flight at the 
deadline");
+        }
+        checkCompletions();
+        finished = true;
+    }
+
+    private void startRoute(DefaultCamelContext context) {
+        try {
+            context.getRouteController().startRoute(ROUTE);
+            ledger.record("routeStarted", "r", ROUTE);
+        } catch (Exception e) {
+            violations.add("start failed: " + e);
+        }
+    }
+
+    private void planSends() {
+        DstExecutor sender = dst.executor("sender", 1);
+        long t = dst.between(0, 500);
+        int n = 0;
+        int bursts = (int) dst.between(1, 4);
+        for (int b = 0; b < bursts; b++) {
+            int size = (int) dst.between(1, 12);
+            for (int i = 0; i < size; i++) {
+                String id = "m" + n++;
+                sendsLeft++;
+                sender.schedule("send " + id, t, () -> {
+                    bus.send(id);
+                    sendsLeft--;
+                });
+                t += dst.between(0, 100);
+            }
+            t += dst.between(0, 30_000);
+        }
+    }
+
+    /**
+     * Stops the route while messages may be processed (the consumer stops 
receiving, the shutdown strategy waits for
+     * the exchanges in flight, whose asynchronous answers are tasks of their 
own), then starts it again.
+     */
+    private void planOperations(DefaultCamelContext context, DstExecutor 
operator) {
+        int ops = (int) dst.between(0, 3);
+        long t = dst.between(500, 5000);
+        for (int i = 0; i < ops; i++) {
+            opsLeft++;
+            operator.schedule("stop route", t, () -> {
+                if (!stopping && 
context.getRouteController().getRouteStatus(ROUTE) == ServiceStatus.Started) {
+                    heldAtStop.addAll(inProgress);
+                    // only the start after this stop ends the stopping state 
(not the start of a skipped stop)
+                    gracefulStop(context, operator, () -> restart(context, 
operator, true));
+                } else {
+                    restart(context, operator, false);
+                }
+            });
+            t += dst.between(5000, 60_000);
+        }
+    }
+
+    private void restart(DefaultCamelContext context, DstExecutor operator, 
boolean afterStop) {
+        operator.schedule("start route", dst.between(100, 10_000), () -> {
+            if (afterStop) {
+                stopping = false;
+            }
+            startRoute(context);
+            opsLeft--;
+        });
+    }
+
+    /**
+     * The graceful stop of the route by DefaultShutdownStrategy as it reaches 
the consumer, with the wait of the
+     * strategy as tasks of the operator: deferShutdown(shutdownRunningTask of 
the route), then a check every second
+     * (the loop delay of the strategy) until the route has no exchange in 
flight and the consumer none pending, for up
+     * to the timeout of the strategy; then stopRoute, which finds nothing to 
wait for and stops the consumer. When the
+     * timeout passes first, the strategy forces the consumer to stop 
(shutdownNowOnTimeout, the default), which this
+     * does with ServiceHelper.stopService before stopRoute. Here stopRoute is 
a second graceful shutdown, which the
+     * real strategy does not make (it stops the route services at once): it 
waits for the exchanges still in flight (on
+     * the one stack its own timeout is noticed only when that wait returns), 
which delays the route stop and the
+     * restart. Nothing is delivered meanwhile, as the stopped consumer has 
closed its processor client, and the
+     * settlements of those exchanges find the client closed as they would 
after the real forced stop. Settlements after
+     * a forced stop are "closed" (by design); a settlement inside the 
graceful wait must find the client open.
+     * <p>
+     * Why not stopRoute at once: the consumer may hold a message delivered 
during the graceful stop in processMessage
+     * until it stops. On the one stack that call would run above the wait of 
the strategy, and the stop below it could
+     * not go on until the call returned.
+     * <p>
+     * This and {@link #awaitInflightThenStop} mirror {@code 
DefaultShutdownStrategy.ShutdownTask} (deferShutdown, the
+     * wait for the inflight exchanges and the pending exchanges of the 
consumer, the forced stop on timeout): check
+     * them against it when the strategy changes.
+     */
+    private void gracefulStop(DefaultCamelContext context, DstExecutor 
operator, Runnable then) {
+        Route route = context.getRoute(ROUTE);
+        ShutdownAware consumer = (ShutdownAware) route.getConsumer();
+        stopping = true;
+        consumer.deferShutdown(route.getShutdownRunningTask());
+        ledger.record("routeStopping", "r", ROUTE);
+        long timeout = 
context.getShutdownStrategy().getTimeUnit().toMillis(context.getShutdownStrategy().getTimeout());
+        awaitInflightThenStop(context, consumer, operator, dst.now() + 
timeout, then);
+    }
+
+    private void awaitInflightThenStop(
+            DefaultCamelContext context, ShutdownAware consumer, DstExecutor 
operator, long deadline, Runnable then) {
+        int pending = context.getInflightRepository().size(ROUTE) + 
Math.max(0, consumer.getPendingExchangesSize());
+        if (pending > 0 && dst.now() < deadline) {
+            operator.schedule("graceful stop: wait for the exchanges in 
flight", 1000,
+                    () -> awaitInflightThenStop(context, consumer, operator, 
deadline, then));
+            return;
+        }
+        try {
+            if (pending > 0) {
+                dst.trace("world: graceful stop timed out with " + pending + " 
exchanges, the consumer is stopped now");
+                forcedAtStop.addAll(inProgress);
+                ServiceHelper.stopService(consumer);
+            }
+            context.getRouteController().stopRoute(ROUTE);
+            ledger.record("routeStopped", "r", ROUTE);
+        } catch (Exception e) {
+            violations.add("stop failed: " + e);
+        }
+        then.run();
+    }
+}
diff --git 
a/components/camel-azure/camel-azure-servicebus/src/test/resources/log4j2.properties
 
b/components/camel-azure/camel-azure-servicebus/src/test/resources/log4j2.properties
index 91c0226629df..d3f9d7de7b11 100644
--- 
a/components/camel-azure/camel-azure-servicebus/src/test/resources/log4j2.properties
+++ 
b/components/camel-azure/camel-azure-servicebus/src/test/resources/log4j2.properties
@@ -14,9 +14,16 @@
 ## See the License for the specific language governing permissions and
 ## limitations under the License.
 ## ---------------------------------------------------------------------------
-appender.file.type = File
+# capped: the DST test runs the consumer for a thousand seeds
+appender.file.type = RollingFile
 appender.file.name = file
 appender.file.fileName = target/camel-servicebus-test.log
+appender.file.filePattern = target/camel-servicebus-test.%i.log
+appender.file.policies.type = Policies
+appender.file.policies.size.type = SizeBasedTriggeringPolicy
+appender.file.policies.size.size = 50MB
+appender.file.strategy.type = DefaultRolloverStrategy
+appender.file.strategy.max = 1
 appender.file.layout.type = PatternLayout
 appender.file.layout.pattern = %d [%-15.15t] %-5p %-30.30c{1} - %m%n
 appender.out.type = Console
diff --git a/components/camel-test/camel-test-dst/src/main/docs/test-dst.adoc 
b/components/camel-test/camel-test-dst/src/main/docs/test-dst.adoc
index 43249aaa404e..c834da4f739a 100644
--- a/components/camel-test/camel-test-dst/src/main/docs/test-dst.adoc
+++ b/components/camel-test/camel-test-dst/src/main/docs/test-dst.adoc
@@ -91,6 +91,16 @@ a|* checkpointing by checkpointBatchSize and 
checkpointBatchTimeout
 * an asynchronous downstream step
 |CAMEL-25683, CAMEL-25738, CAMEL-25781
 
+|camel-azure-servicebus
+|`ServiceBusWorld` (`ServiceBusConsumerDstTest`)
+|FakeServiceBus: a queue in peek-lock mode (lock duration, delivery count, 
dead-letter queue) behind the processor and receiver clients of 
ServiceBusClientFactory
+a|* the lock renewer of the consumer (PEEK_LOCK, maxAutoLockRenewDuration) 
with an asynchronous route
+* processing that takes longer than the lock duration
+* complete, abandon and dead-letter, also after the lock was lost
+* maxConcurrentCalls of one to three (with the asynchronous route it bounds 
the processMessage calls)
+* route stop and start while messages are processed
+|CAMEL-25701, CAMEL-26956, CAMEL-26969, CAMEL-26970, CAMEL-27066
+
 |camel-google-pubsub
 |`PubsubWorld` (`PubsubConsumerDstTest`)
 |FakePubsub: a subscription (Pull, Acknowledge, ModifyAckDeadline, ack 
deadlines, dead letter policy) behind the SubscriberStub of synchronousPull

Reply via email to