This is an automated email from the ASF dual-hosted git repository.

davsclaus pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/camel.git


The following commit(s) were added to refs/heads/main by this push:
     new 6e5fe2671f9f CAMEL-26958: camel-paho-mqtt5 - a DST fake world for the 
consumer (#30138)
6e5fe2671f9f is described below

commit 6e5fe2671f9f7ffc7418017b184295f3917472d9
Author: allthingssecurity <[email protected]>
AuthorDate: Tue Nov 17 03:27:20 2026 +0530

    CAMEL-26958: camel-paho-mqtt5 - a DST fake world for the consumer (#30138)
    
    Co-Authored-By: Claude Opus 5.5 <[email protected]>
---
 .../camel/catalog/components/paho-mqtt5.json       |   3 +
 .../org/apache/camel/catalog/docs/test-dst.adoc    |  16 +
 components/camel-paho-mqtt5/pom.xml                |  26 +
 .../camel/component/paho/mqtt5/paho-mqtt5.json     |   3 +
 .../paho/mqtt5/PahoMqtt5DstComponent.java          |  62 ++
 .../camel/component/paho/mqtt5/dst/FakeMqtt.java   | 741 +++++++++++++++++++++
 .../paho/mqtt5/dst/PahoMqtt5ConsumerDstTest.java   |  37 +
 .../component/paho/mqtt5/dst/PahoMqtt5World.java   | 480 +++++++++++++
 .../src/test/resources/log4j2.properties           |   9 +-
 .../camel-test-dst/src/main/docs/test-dst.adoc     |  16 +
 10 files changed, 1392 insertions(+), 1 deletion(-)

diff --git 
a/catalog/camel-catalog/src/generated/resources/org/apache/camel/catalog/components/paho-mqtt5.json
 
b/catalog/camel-catalog/src/generated/resources/org/apache/camel/catalog/components/paho-mqtt5.json
index 166e99559579..7a5eb8b61ae7 100644
--- 
a/catalog/camel-catalog/src/generated/resources/org/apache/camel/catalog/components/paho-mqtt5.json
+++ 
b/catalog/camel-catalog/src/generated/resources/org/apache/camel/catalog/components/paho-mqtt5.json
@@ -106,5 +106,8 @@
     "sslClientProps": { "index": 31, "kind": "parameter", "displayName": "Ssl 
Client Props", "group": "security", "label": "security", "required": false, 
"type": "object", "javaType": "java.util.Properties", "deprecated": false, 
"autowired": false, "secret": false, "configurationClass": 
"org.apache.camel.component.paho.mqtt5.PahoMqtt5Configuration", 
"configurationField": "configuration", "description": "Sets the SSL properties 
for the connection. Note that these properties are only valid [...]
     "sslHostnameVerifier": { "index": 32, "kind": "parameter", "displayName": 
"Ssl Hostname Verifier", "group": "security", "label": "security", "required": 
false, "type": "object", "javaType": "javax.net.ssl.HostnameVerifier", 
"deprecated": false, "autowired": false, "secret": false, "configurationClass": 
"org.apache.camel.component.paho.mqtt5.PahoMqtt5Configuration", 
"configurationField": "configuration", "description": "Sets the 
HostnameVerifier for the SSL connection. Note that it wi [...]
     "userName": { "index": 33, "kind": "parameter", "displayName": "User 
Name", "group": "security", "label": "security", "required": false, "type": 
"string", "javaType": "java.lang.String", "deprecated": false, "autowired": 
false, "secret": true, "security": "secret", "configurationClass": 
"org.apache.camel.component.paho.mqtt5.PahoMqtt5Configuration", 
"configurationField": "configuration", "description": "Username to be used for 
authentication against the MQTT broker" }
+  },
+  "dstWorlds": {
+    "PahoMqtt5World": { "javaType": 
"org.apache.camel.component.paho.mqtt5.dst.PahoMqtt5World", "fake": "FakeMqtt: 
an MQTT 5 broker with the session of the client id (Clean Start, Session 
Expiry, Receive Maximum, resend on reconnect) and a Paho MqttClient subclass on 
it, given to the consumer by a test component", "covers": [ "QoS 0, 1 and 2 
with automatic or manual acknowledgements (manualAcksEnabled)", "a session that 
the broker keeps (cleanStart=false, sessionExpiryInterval) or not",  [...]
   }
 }
diff --git 
a/catalog/camel-catalog/src/generated/resources/org/apache/camel/catalog/docs/test-dst.adoc
 
b/catalog/camel-catalog/src/generated/resources/org/apache/camel/catalog/docs/test-dst.adoc
index 4923ed2b2935..43249aaa404e 100644
--- 
a/catalog/camel-catalog/src/generated/resources/org/apache/camel/catalog/docs/test-dst.adoc
+++ 
b/catalog/camel-catalog/src/generated/resources/org/apache/camel/catalog/docs/test-dst.adoc
@@ -134,6 +134,16 @@ a|* the JetStream pull consumer with explicit acks and 
naks, ephemeral or durabl
 * a server restart that deletes the ephemeral consumer, and the subscription 
made again
 |CAMEL-26884
 
+|camel-paho-mqtt5
+|`PahoMqtt5World` (`PahoMqtt5ConsumerDstTest`)
+|FakeMqtt: an MQTT 5 broker with the session of the client id (Clean Start, 
Session Expiry, Receive Maximum, resend on reconnect) and a Paho MqttClient 
subclass on it, given to the consumer by a test component
+a|* QoS 0, 1 and 2 with automatic or manual acknowledgements 
(manualAcksEnabled)
+* a session that the broker keeps (cleanStart=false, sessionExpiryInterval) or 
not
+* the automatic reconnect of Paho and the subscription made again
+* the reconnect of the consumer after a failed message, and Receive Maximum
+* route stop and start, with messages queued for a kept session
+|CAMEL-25793, CAMEL-25944, CAMEL-27041, CAMEL-27043
+
 |camel-pulsar
 |`PulsarWorld` (`PulsarConsumerDstTest`)
 |FakePulsar: a broker and client for one topic and one durable subscription
@@ -615,6 +625,12 @@ CAMEL-25737), `dst.replayCheckSeed` (the seed of the 
in-JVM replay test of the E
 `dst.share.pollOnError` (the pollOnError strategies the Kafka share world 
picks from)
 and `dst.pulsar.subscribeFailRates` (the subscribe failure rates of the Pulsar 
world, `0` runs without subscribe failures).
 
+For camel-paho-mqtt5 (`components/camel-paho-mqtt5`, 
`PahoMqtt5ConsumerDstTest`) the past fixes are the main code of
+6044f38166f9 (CAMEL-25944) and the change of b7594f06e0b7 (CAMEL-25793), taken 
out by hand. Its world has two
+properties, each adding back a case that the default set leaves out: 
`dst.paho.strictCallbackThread` (`true` checks a
+client whose connect hangs while the callback thread is still in 
messageArrived, CAMEL-27043) and
+`dst.paho.stopWhileConnecting` (`true` lets the operator stop the route while 
the client connects, CAMEL-27041).
+
 == Limits
 
 * Interleaving is per task: the harness does not preempt a task in the middle 
of a method.
diff --git a/components/camel-paho-mqtt5/pom.xml 
b/components/camel-paho-mqtt5/pom.xml
index fac0b61552b2..4fdae28dd234 100644
--- a/components/camel-paho-mqtt5/pom.xml
+++ b/components/camel-paho-mqtt5/pom.xml
@@ -34,6 +34,8 @@
 
     <properties>
         <skipITs.s390x>true</skipITs.s390x>
+        <!-- the DST tests load the byte code agent of camel-test-dst when the 
JVM starts (see test-dst.adoc) -->
+        
<camel.surefire.fork.additional-vmargs>"-javaagent:${net.bytebuddy:byte-buddy-agent:jar}"</camel.surefire.fork.additional-vmargs>
     </properties>
 
     <dependencies>
@@ -77,6 +79,30 @@
             <version>${project.version}</version>
             <scope>test</scope>
         </dependency>
+        <!-- deterministic simulation test of the consumer (dst package) -->
+        <dependency>
+            <groupId>org.apache.camel</groupId>
+            <artifactId>camel-test-dst</artifactId>
+            <scope>test</scope>
+        </dependency>
     </dependencies>
 
+    <build>
+        <plugins>
+            <plugin>
+                <!-- sets the property net.bytebuddy:byte-buddy-agent:jar to 
the path of the agent of the DST tests -->
+                <groupId>org.apache.maven.plugins</groupId>
+                <artifactId>maven-dependency-plugin</artifactId>
+                <executions>
+                    <execution>
+                        <id>dst-agent-path</id>
+                        <goals>
+                            <goal>properties</goal>
+                        </goals>
+                    </execution>
+                </executions>
+            </plugin>
+        </plugins>
+    </build>
+
 </project>
diff --git 
a/components/camel-paho-mqtt5/src/generated/resources/META-INF/org/apache/camel/component/paho/mqtt5/paho-mqtt5.json
 
b/components/camel-paho-mqtt5/src/generated/resources/META-INF/org/apache/camel/component/paho/mqtt5/paho-mqtt5.json
index 166e99559579..7a5eb8b61ae7 100644
--- 
a/components/camel-paho-mqtt5/src/generated/resources/META-INF/org/apache/camel/component/paho/mqtt5/paho-mqtt5.json
+++ 
b/components/camel-paho-mqtt5/src/generated/resources/META-INF/org/apache/camel/component/paho/mqtt5/paho-mqtt5.json
@@ -106,5 +106,8 @@
     "sslClientProps": { "index": 31, "kind": "parameter", "displayName": "Ssl 
Client Props", "group": "security", "label": "security", "required": false, 
"type": "object", "javaType": "java.util.Properties", "deprecated": false, 
"autowired": false, "secret": false, "configurationClass": 
"org.apache.camel.component.paho.mqtt5.PahoMqtt5Configuration", 
"configurationField": "configuration", "description": "Sets the SSL properties 
for the connection. Note that these properties are only valid [...]
     "sslHostnameVerifier": { "index": 32, "kind": "parameter", "displayName": 
"Ssl Hostname Verifier", "group": "security", "label": "security", "required": 
false, "type": "object", "javaType": "javax.net.ssl.HostnameVerifier", 
"deprecated": false, "autowired": false, "secret": false, "configurationClass": 
"org.apache.camel.component.paho.mqtt5.PahoMqtt5Configuration", 
"configurationField": "configuration", "description": "Sets the 
HostnameVerifier for the SSL connection. Note that it wi [...]
     "userName": { "index": 33, "kind": "parameter", "displayName": "User 
Name", "group": "security", "label": "security", "required": false, "type": 
"string", "javaType": "java.lang.String", "deprecated": false, "autowired": 
false, "secret": true, "security": "secret", "configurationClass": 
"org.apache.camel.component.paho.mqtt5.PahoMqtt5Configuration", 
"configurationField": "configuration", "description": "Username to be used for 
authentication against the MQTT broker" }
+  },
+  "dstWorlds": {
+    "PahoMqtt5World": { "javaType": 
"org.apache.camel.component.paho.mqtt5.dst.PahoMqtt5World", "fake": "FakeMqtt: 
an MQTT 5 broker with the session of the client id (Clean Start, Session 
Expiry, Receive Maximum, resend on reconnect) and a Paho MqttClient subclass on 
it, given to the consumer by a test component", "covers": [ "QoS 0, 1 and 2 
with automatic or manual acknowledgements (manualAcksEnabled)", "a session that 
the broker keeps (cleanStart=false, sessionExpiryInterval) or not",  [...]
   }
 }
diff --git 
a/components/camel-paho-mqtt5/src/test/java/org/apache/camel/component/paho/mqtt5/PahoMqtt5DstComponent.java
 
b/components/camel-paho-mqtt5/src/test/java/org/apache/camel/component/paho/mqtt5/PahoMqtt5DstComponent.java
new file mode 100644
index 000000000000..cfaafd4ad6fe
--- /dev/null
+++ 
b/components/camel-paho-mqtt5/src/test/java/org/apache/camel/component/paho/mqtt5/PahoMqtt5DstComponent.java
@@ -0,0 +1,62 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.camel.component.paho.mqtt5;
+
+import java.util.Map;
+import java.util.function.Supplier;
+
+import org.apache.camel.CamelContext;
+import org.apache.camel.Consumer;
+import org.apache.camel.Endpoint;
+import org.apache.camel.Processor;
+import org.eclipse.paho.mqttv5.client.MqttClient;
+
+/**
+ * The paho-mqtt5 component whose consumers get their own client from a 
supplier instead of creating a Paho client (the
+ * package-private {@code PahoMqtt5Consumer.createClient()}), for the DST 
world of the consumer (dst package). The
+ * endpoint and the consumer are the real classes. Two differences from the 
real component: the world sets the options
+ * on the configuration of the component and the endpoint uri has none (this 
component and its endpoint are not the
+ * annotated classes that the build generates configurers for), and there is 
no client option (a client given to the
+ * endpoint is not passed on).
+ */
+public final class PahoMqtt5DstComponent extends PahoMqtt5Component {
+
+    private final Supplier<MqttClient> clients;
+
+    public PahoMqtt5DstComponent(CamelContext context, Supplier<MqttClient> 
clients) {
+        super(context);
+        this.clients = clients;
+    }
+
+    @Override
+    protected Endpoint createEndpoint(String uri, String remaining, 
Map<String, Object> parameters) throws Exception {
+        // the options come from the configuration of the component 
(setConfiguration); the endpoint uri has none
+        return new PahoMqtt5Endpoint(uri, remaining, this, 
getConfiguration().copy()) {
+            @Override
+            public Consumer createConsumer(Processor processor) throws 
Exception {
+                PahoMqtt5Consumer consumer = new PahoMqtt5Consumer(this, 
processor) {
+                    @Override
+                    MqttClient createClient() {
+                        return clients.get();
+                    }
+                };
+                configureConsumer(consumer);
+                return consumer;
+            }
+        };
+    }
+}
diff --git 
a/components/camel-paho-mqtt5/src/test/java/org/apache/camel/component/paho/mqtt5/dst/FakeMqtt.java
 
b/components/camel-paho-mqtt5/src/test/java/org/apache/camel/component/paho/mqtt5/dst/FakeMqtt.java
new file mode 100644
index 000000000000..c54638c47c7c
--- /dev/null
+++ 
b/components/camel-paho-mqtt5/src/test/java/org/apache/camel/component/paho/mqtt5/dst/FakeMqtt.java
@@ -0,0 +1,741 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.camel.component.paho.mqtt5.dst;
+
+import java.lang.reflect.Proxy;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.function.BooleanSupplier;
+import java.util.function.Consumer;
+
+import org.apache.camel.test.dst.Dst;
+import org.apache.camel.test.dst.DstExecutor;
+import org.eclipse.paho.mqttv5.client.IMqttToken;
+import org.eclipse.paho.mqttv5.client.MqttCallback;
+import org.eclipse.paho.mqttv5.client.MqttClient;
+import org.eclipse.paho.mqttv5.client.MqttClientException;
+import org.eclipse.paho.mqttv5.client.MqttConnectionOptions;
+import org.eclipse.paho.mqttv5.client.MqttDisconnectResponse;
+import org.eclipse.paho.mqttv5.client.persist.MemoryPersistence;
+import org.eclipse.paho.mqttv5.common.MqttException;
+import org.eclipse.paho.mqttv5.common.MqttMessage;
+
+/**
+ * A fake MQTT 5 broker with one topic and the session of the consumer's 
client id, and the Paho client of the consumer
+ * ({@link Client}, a subclass of the Paho {@link MqttClient} whose network 
calls go to the fake). Written from the MQTT
+ * 5.0 specification (OASIS) and Paho 1.2.5; the rules it follows are quoted 
where they are implemented and marked:
+ * <ul>
+ * <li>[S] MQTT Version 5.0, OASIS Standard: 3.1.2.4 Clean Start, 3.1.2.11.2 
Session Expiry Interval, 3.2.2.1.1 Session
+ * Present, 3.3.4 PUBLISH actions, 4.4 Message delivery retry, 4.9 Flow 
Control (Receive Maximum);</li>
+ * <li>[P] Paho Java client 1.2.5 (MqttClient, MqttAsyncClient and 
CommsCallback, read with javap): callbacks on one
+ * callback thread, manual acknowledgements, the automatic reconnect;</li>
+ * <li>[A] an assumption, where the documents leave the behaviour open; the 
seed picks between the allowed choices.</li>
+ * </ul>
+ * <p>
+ * Modelled: the session of the client id (subscriptions, QoS 1 messages sent 
and not acknowledged, messages queued
+ * while the client is offline), Clean Start, the Session Expiry Interval, 
Session Present, the resend of unacknowledged
+ * QoS 1 messages on a reconnect to a kept session, Receive Maximum, a 
connection lost by the broker (and the automatic
+ * reconnect of Paho, with its delay doubled from 1 second up to 
maxReconnectDelay), a broker that cannot be reached for
+ * a while, a connection lost during a subscribe, QoS 0, 1 and 2.
+ * <p>
+ * Modelled from Paho as well: a message that arrives while no callback is set 
is dropped (CommsCallback), the callback
+ * thread that a disconnect or a lost connection stops and that the next 
connect starts again only when it has stopped
+ * (a connect that comes while the thread is still in messageArrived hangs: 
CommsCallback.start waits for a thread that
+ * does not start again, and the client stays connecting), a connect or a 
close while a connect is in progress fails
+ * (32110), and a PUBACK of a disconnected client is dropped.
+ * <p>
+ * Not modelled: retained messages, wildcards and several topics, shared 
subscriptions, will messages, the QoS 2 flow
+ * beyond PUBREC (Paho sends PUBREC when the message arrives and completes the 
flow on its own, so the broker never
+ * sends a QoS 2 message again once the client has it; the fake frees the 
Receive Maximum slot of a QoS 2 message at
+ * PUBREC, one round trip before PUBCOMP, where MQTT 5 4.9 returns it), 
message expiry, topic aliases, keep alive, the
+ * persistence of the client, the quiesce of a disconnect, the bound of 10 
messages of the queue of the callback thread,
+ * the reconnect delay that Paho keeps for all clients of the JVM (each client 
has its own here), the order of
+ * connectComplete and the messages resent at the connect (separate tasks 
here), the threads that Paho calls
+ * connectComplete and disconnected on (tasks of the callback executor here), 
and the reason codes of a SUBACK.
+ * <p>
+ * Timing (the one-stack rule of the harness): every call of the client takes 
a round trip and writes take effect at its
+ * start; deliveries are tasks of the callback thread, one at a time, in order.
+ */
+final class FakeMqtt {
+
+    static final String TOPIC = "dst/orders";
+    static final String URI = "tcp://fake:1883";
+
+    private final Dst dst;
+    final List<String> violations = new ArrayList<>();
+    boolean faults = true;
+    double subscribeFailRate;
+    double slowRate;
+    /** [A] whether the broker keeps QoS 0 messages for an offline session 
(MQTT allows it, 3.1.2.4). */
+    boolean queueQos0;
+
+    private final DstExecutor callbackThread;
+    private final DstExecutor reconnectThread;
+    private Session session;
+    private Client connected;
+    private long unreachableUntil;
+    private final List<Client> clients = new ArrayList<>();
+    /** The QoS 1 and 2 messages that the broker accepted for a session that 
ended before it delivered them. */
+    final List<Long> lostWithSession = new ArrayList<>();
+    /**
+     * The connects that came while the callback thread of the client was 
still in messageArrived after the connection
+     * closed: they hang (see Client.connectToBroker), until the world ends 
the run with releaseHung.
+     */
+    int hungConnects;
+    boolean releaseHung;
+
+    /** Whether a client connects (its connect or its automatic reconnect is 
in progress), other than a hung one. */
+    boolean anyConnecting() {
+        return clients.stream().anyMatch(c -> c.isConnecting() && !c.isHung());
+    }
+
+    /** Whether a client hangs in a connect now (until the world ends the run 
with releaseHung). */
+    boolean anyHanging() {
+        return clients.stream().anyMatch(c -> c.isHung() && c.isConnecting());
+    }
+
+    /** Whether the client the consumer made last hangs in a connect. */
+    boolean currentHung() {
+        return !clients.isEmpty() && clients.get(clients.size() - 1).isHung();
+    }
+
+    private void reachable() throws MqttException {
+        if (dst.now() < unreachableUntil) {
+            throw new 
MqttException(MqttClientException.REASON_CODE_SERVER_CONNECT_ERROR);
+        }
+    }
+
+    /**
+     * The QoS 0 and 2 messages that the client received and dropped from its 
queue: when a connect cleared it, or when
+     * the client was closed.
+     */
+    final List<Long> lostInClient = new ArrayList<>();
+    Consumer<Long> onAck = n -> {
+    };
+    Consumer<Long> onDropped = n -> {
+    };
+    Consumer<Long> onDelivered = n -> {
+    };
+    Runnable onSubscribeFailed = () -> {
+    };
+    Runnable onSubscribed = () -> {
+    };
+    /** Whether the connection may be lost during a subscribe now (see the 
world). */
+    BooleanSupplier subscribeFaultAllowed = () -> true;
+
+    /** A message of the topic. */
+    static final class Msg {
+        final long n;
+        final int qos;
+
+        Msg(long n, int qos) {
+            this.n = n;
+            this.qos = qos;
+        }
+    }
+
+    /** [S] 4.1 the session state of the server. */
+    private final class Session {
+        int subscribedQos = -1;
+        final List<Msg> queued = new ArrayList<>();
+        final Map<Integer, Msg> inflight = new LinkedHashMap<>();
+        long expiryInterval;
+        long expiresAt = Long.MAX_VALUE;
+        int nextPacketId = 1;
+
+        int packetId() {
+            int id = nextPacketId;
+            do {
+                nextPacketId = nextPacketId % 65535 + 1;
+            } while (inflight.containsKey(nextPacketId));
+            return id;
+        }
+    }
+
+    FakeMqtt(Dst dst) {
+        this.dst = dst;
+        this.callbackThread = dst.executor("paho-callback", 1);
+        this.reconnectThread = dst.executor("paho-reconnect", 1);
+    }
+
+    /** A new Paho client, as PahoMqtt5Consumer.createClient makes one. */
+    MqttClient newClient() {
+        try {
+            Client c = new Client();
+            clients.add(c);
+            return c;
+        } catch (MqttException e) {
+            throw new IllegalStateException(e);
+        }
+    }
+
+    List<Client> clients() {
+        return clients;
+    }
+
+    // 
------------------------------------------------------------------------------------------------------------
+    // the publisher and the operator
+
+    /** [S] 3.3.4: the broker sends the message to the subscribed session, at 
the lower of the two QoS. */
+    void publish(long n, int qos) {
+        expireSession();
+        if (session == null || session.subscribedQos < 0) {
+            dst.trace("mqtt: message " + n + " has no subscriber");
+            return;
+        }
+        Msg m = new Msg(n, Math.min(qos, session.subscribedQos));
+        if (m.qos == 0 && connected == null && !queueQos0) {
+            dst.trace("mqtt: QoS 0 message " + n + " dropped, the client is 
offline");
+            return;
+        }
+        session.queued.add(m);
+        pump();
+    }
+
+    /** The broker closes the network connection of the client (a broker 
restart, a network fault). */
+    void dropConnection() {
+        Client c = connected;
+        if (c != null) {
+            dst.trace("mqtt: connection of the client lost");
+            closeConnection(c);
+            c.connectionLost();
+        }
+    }
+
+    /** The broker cannot be reached for {@code millis}. */
+    void unreachable(long millis) {
+        unreachableUntil = Math.max(unreachableUntil, dst.now() + millis);
+        dst.trace("mqtt: broker unreachable until t=" + unreachableUntil);
+        dropConnection();
+    }
+
+    /** The QoS 1 messages sent to the client and not acknowledged, and the 
messages queued for it. */
+    int pending() {
+        expireSession();
+        return session == null ? 0 : session.inflight.size() + 
session.queued.size();
+    }
+
+    boolean isConnected() {
+        return connected != null;
+    }
+
+    boolean isSubscribed() {
+        expireSession();
+        return session != null && session.subscribedQos >= 0;
+    }
+
+    // 
------------------------------------------------------------------------------------------------------------
+    // the broker
+
+    private void expireSession() {
+        if (session != null && connected == null && dst.now() >= 
session.expiresAt) {
+            dst.trace("mqtt: session expired");
+            endSession();
+        }
+    }
+
+    private void endSession() {
+        for (Msg m : session.inflight.values()) {
+            lostWithSession.add(m.n);
+        }
+        for (Msg m : session.queued) {
+            if (m.qos > 0) {
+                lostWithSession.add(m.n);
+            }
+        }
+        session = null;
+    }
+
+    /** [S] 4.9 the server sends no more than Receive Maximum QoS 1 and 2 
PUBLISH packets that are not acknowledged. */
+    private void pump() {
+        Client c = connected;
+        if (c == null || session == null) {
+            return;
+        }
+        while (!session.queued.isEmpty()) {
+            Msg m = session.queued.get(0);
+            if (m.qos > 0 && session.inflight.size() >= c.receiveMaximum) {
+                return;
+            }
+            session.queued.remove(0);
+            int id = 0;
+            if (m.qos > 0) {
+                // [S] 4.3.2 / 4.3.3: a QoS 1 message is in flight until its 
PUBACK, a QoS 2 message until the client
+                // has it (PUBREC; Paho then completes the flow on its own)
+                id = session.packetId();
+                session.inflight.put(id, m);
+            }
+            c.deliver(m, id, false);
+        }
+    }
+
+    private boolean connect(Client c, MqttConnectionOptions options) throws 
MqttException {
+        reachable();
+        if (connected != null && connected != c) {
+            // [S] 3.1.4: a second connection with the same client id makes 
the server disconnect the first one
+            Client old = connected;
+            closeConnection(old);
+            old.connectionLost();
+        }
+        expireSession();
+        boolean present;
+        if (options.isCleanStart() || session == null) {
+            // [S] 3.1.2.4: Clean Start 1 discards any existing session and 
starts a new one
+            if (session != null) {
+                endSession();
+            }
+            session = new Session();
+            present = false;
+        } else {
+            present = true;
+        }
+        Long expiry = options.getSessionExpiryInterval();
+        // [S] 3.1.2.11.2: an absent Session Expiry Interval is 0, the session 
ends when the connection closes
+        session.expiryInterval = expiry == null ? 0 : expiry;
+        session.expiresAt = Long.MAX_VALUE;
+        connected = c;
+        c.sessionPresent = present;
+        dst.trace("mqtt: connected, session present=" + present);
+        // [S] 4.4: "When a Client reconnects with Clean Start set to 0 and a 
session is present, both the Client and
+        // Server MUST resend any unacknowledged PUBLISH packets (where QoS > 
0) and PUBREL packets using their original
+        // Packet Identifiers"
+        for (Map.Entry<Integer, Msg> e : session.inflight.entrySet()) {
+            c.deliver(e.getValue(), e.getKey(), true);
+        }
+        pump();
+        return present;
+    }
+
+    private void closeConnection(Client c) {
+        if (connected != c) {
+            return;
+        }
+        connected = null;
+        c.epoch++;
+        c.connectedFlag = false;
+        c.stopCallback();
+        if (session != null) {
+            if (session.expiryInterval == 0) {
+                endSession();
+            } else {
+                session.expiresAt = dst.now() + session.expiryInterval * 1000;
+            }
+        }
+    }
+
+    private long latency() {
+        if (faults && dst.chance(slowRate)) {
+            return dst.between(200, 3000);
+        }
+        return dst.between(1, 20);
+    }
+
+    // 
------------------------------------------------------------------------------------------------------------
+    // the Paho client
+
+    /** The Paho MqttClient of the consumer: its network calls go to the fake 
broker. */
+    final class Client extends MqttClient {
+        private MqttCallback callback;
+        private boolean manualAcks;
+        private boolean closed;
+        boolean connectedFlag;
+        boolean sessionPresent;
+        int epoch;
+        int receiveMaximum = 65535;
+        private MqttConnectionOptions options;
+        private boolean reconnectCycle;
+        // [P] MqttAsyncClient.reconnecting: set at the first lost connection 
with automaticReconnect
+        private boolean reconnecting;
+        private boolean disconnectedByUser;
+        private boolean connecting;
+        private boolean hung;
+        // [P] CommsCallback: the callback thread, its target and current 
state, and its queue of received messages
+        private boolean callbackTarget;
+        private boolean callbackRunning;
+        private boolean busy;
+        private boolean workerScheduled;
+        private final List<Object[]> inbox = new ArrayList<>();
+
+        Client() throws MqttException {
+            super(URI, "dst-consumer", new MemoryPersistence());
+        }
+
+        private void checkOpen() throws MqttException {
+            if (closed) {
+                violations.add("a call on a closed client");
+                throw new 
MqttException(MqttClientException.REASON_CODE_CLIENT_CLOSED);
+            }
+        }
+
+        @Override
+        public void setCallback(MqttCallback callback) {
+            this.callback = callback;
+        }
+
+        @Override
+        public void setManualAcks(boolean manualAcks) {
+            this.manualAcks = manualAcks;
+        }
+
+        @Override
+        public boolean isConnected() {
+            return connectedFlag;
+        }
+
+        @Override
+        public void connect(MqttConnectionOptions options) throws 
MqttException {
+            connectWithResult(options);
+        }
+
+        @Override
+        public IMqttToken connectWithResult(MqttConnectionOptions options) 
throws MqttException {
+            checkOpen();
+            if (connecting) {
+                // [P] MqttAsyncClient.connect: a connect while one is in 
progress fails
+                throw new 
MqttException(MqttClientException.REASON_CODE_CONNECT_IN_PROGRESS);
+            }
+            if (connectedFlag) {
+                throw new 
MqttException(MqttClientException.REASON_CODE_CLIENT_CONNECTED);
+            }
+            this.options = options;
+            Integer rm = options.getReceiveMaximum();
+            receiveMaximum = rm == null ? 65535 : rm;
+            connecting = true;
+            boolean present;
+            try {
+                dst.block(latency(), "connect");
+                present = connectToBroker();
+            } finally {
+                connecting = false;
+            }
+            connectedFlag = true;
+            disconnectedByUser = false;
+            // [P] once a connection was lost with automaticReconnect, a 
connect reports reconnect=true: the reconnecting
+            // flag that MqttReconnectCallback.disconnected sets is not 
cleared (MqttAsyncClient 1.2.5)
+            boolean reconnect = reconnecting;
+            callbackTask("connectComplete", () -> 
callback.connectComplete(reconnect, URI));
+            return token(present);
+        }
+
+        @Override
+        public IMqttToken subscribe(String topic, int qos) throws 
MqttException {
+            checkOpen();
+            if (!connectedFlag) {
+                throw new 
MqttException(MqttClientException.REASON_CODE_CLIENT_NOT_CONNECTED);
+            }
+            if (!TOPIC.equals(topic)) {
+                violations.add("subscribe to an unexpected topic " + topic);
+            }
+            int e = epoch;
+            dst.block(latency(), "subscribe");
+            // [P] MqttClient waits for the SUBACK without a time limit 
(timeToWait -1): the call fails only when the
+            // connection is lost while it waits; [A] the broker closes the 
connection now and then during a subscribe
+            if (e == epoch && faults && subscribeFaultAllowed.getAsBoolean() 
&& dst.chance(subscribeFailRate)) {
+                dst.trace("mqtt: the connection is lost during a subscribe");
+                dropConnection();
+            }
+            if (e != epoch) {
+                onSubscribeFailed.run();
+                throw new 
MqttException(MqttClientException.REASON_CODE_CONNECTION_LOST);
+            }
+            session.subscribedQos = qos;
+            onSubscribed.run();
+            dst.trace("mqtt: subscribed with QoS " + qos);
+            pump();
+            return token(false);
+        }
+
+        @Override
+        public void unsubscribe(String topic) throws MqttException {
+            checkOpen();
+            if (!connectedFlag) {
+                throw new 
MqttException(MqttClientException.REASON_CODE_CLIENT_NOT_CONNECTED);
+            }
+            if (session != null) {
+                session.subscribedQos = -1;
+            }
+            dst.block(latency(), "unsubscribe");
+        }
+
+        @Override
+        public void disconnect() throws MqttException {
+            checkOpen();
+            if (callbackThread.label().equals(dst.currentExecutor())) {
+                // [P] a disconnect from a callback is not allowed
+                violations.add("disconnect from the callback thread");
+                throw new 
MqttException(MqttClientException.REASON_CODE_CLIENT_DISCONNECT_PROHIBITED);
+            }
+            if (!connectedFlag) {
+                throw new 
MqttException(MqttClientException.REASON_CODE_CLIENT_ALREADY_DISCONNECTED);
+            }
+            disconnectedByUser = true;
+            reconnectCycle = false;
+            closeConnection(this);
+            dst.block(latency(), "disconnect");
+        }
+
+        boolean isClosed() {
+            return closed;
+        }
+
+        boolean isConnecting() {
+            return connecting;
+        }
+
+        @Override
+        public void close(boolean force) throws MqttException {
+            if (connecting) {
+                // [P] ClientComms.close: a close while a connect is in 
progress fails, also with force
+                throw new 
MqttException(MqttClientException.REASON_CODE_CONNECT_IN_PROGRESS);
+            }
+            if (connectedFlag) {
+                // [P] close of a connected client fails
+                throw new 
MqttException(MqttClientException.REASON_CODE_CLIENT_CONNECTED);
+            }
+            closed = true;
+            reconnectCycle = false;
+            // the messages still in the queue of the callback thread are gone 
with the client (a QoS 1 message stays in
+            // flight at the broker; a QoS 0 or QoS 2 message is lost)
+            for (Object[] item : inbox) {
+                Msg m = (Msg) item[0];
+                if (m.qos != 1) {
+                    lostInClient.add(m.n);
+                }
+            }
+            inbox.clear();
+        }
+
+        @Override
+        public void close() throws MqttException {
+            close(false);
+        }
+
+        @Override
+        public void messageArrivedComplete(int id, int qos) throws 
MqttException {
+            dst.trace("mqtt: messageArrivedComplete(" + id + ", " + qos + ")");
+            checkOpen();
+            if (qos != 1) {
+                // [P] QoS 2: Paho completed the flow on its own; QoS 0 has 
nothing to acknowledge
+                return;
+            }
+            if (!connectedFlag) {
+                // [P] the PUBACK is queued and dropped with the closed 
connection
+                dst.trace("mqtt: PUBACK for packet id " + id + " dropped, the 
client is disconnected");
+                return;
+            }
+            ack(id);
+        }
+
+        private void ack(int id) {
+            Msg m = session != null ? session.inflight.remove(id) : null;
+            if (m == null) {
+                // [A] the broker ignores a PUBACK for a packet id that has no 
message in flight
+                dst.trace("mqtt: PUBACK for packet id " + id + " ignored, no 
message in flight");
+                return;
+            }
+            dst.trace("mqtt: message " + m.n + " acknowledged");
+            onAck.accept(m.n);
+            pump();
+        }
+
+        /**
+         * [P] the client receives a PUBLISH: 
CommsCallback.messageArrived(MqttPublish) runs on the receiver thread.
+         * Without a callback set at that moment the message is not queued and 
no PUBACK is sent (a QoS 1 message stays
+         * in flight at the broker); a QoS 2 message gets its PUBREC all the 
same (ClientState.notifyReceivedMsg), so
+         * the broker never sends it again. Otherwise it goes to the queue of 
the callback thread.
+         */
+        void deliver(Msg m, int id, boolean dup) {
+            if (m.qos == 2 && session != null) {
+                session.inflight.remove(id);
+            }
+            if (callback == null) {
+                dst.trace("mqtt: message " + m.n + " dropped, no callback");
+                onDropped.accept(m.n);
+            } else {
+                inbox.add(new Object[] { m, id, dup, epoch });
+                runCallback();
+            }
+            if (m.qos == 2) {
+                pump();
+            }
+        }
+
+        /**
+         * [P] ConnectBG: the TCP connect (it fails when the broker cannot be 
reached), then CommsCallback.start, then
+         * the CONNECT. start() restarts the thread only when it is STOPPED, 
and then waits until it runs; a thread that
+         * is still in messageArrived after the connection closed is not 
STOPPED, so start() sets nothing and its wait
+         * never ends: the connect hangs, and the client stays connecting (a 
close fails with 32110). The fake waits
+         * until the world ends the run.
+         */
+        private boolean connectToBroker() throws MqttException {
+            reachable();
+            if (callbackRunning && !callbackTarget) {
+                hung = true;
+                hungConnects++;
+                dst.trace("mqtt: connect while the callback thread is still in 
messageArrived: the connect hangs");
+                dst.awaitUntil(() -> releaseHung, 24L * 3600 * 1000 - 1, 
"connect of a client whose callback thread does not"
+                                                                         + " 
start again");
+                throw new 
MqttException(MqttClientException.REASON_CODE_CLIENT_TIMEOUT);
+            }
+            startCallback();
+            return FakeMqtt.this.connect(this, options);
+        }
+
+        boolean isHung() {
+            return hung;
+        }
+
+        /** [P] CommsCallback.start: restarts the thread only when it is 
STOPPED. */
+        private void startCallback() {
+            if (!callbackRunning) {
+                // [P] start() clears the queue: the messages of an earlier 
connection that were not delivered are gone
+                for (Object[] item : inbox) {
+                    Msg m = (Msg) item[0];
+                    if (m.qos != 1) {
+                        lostInClient.add(m.n);
+                    }
+                }
+                inbox.clear();
+                callbackRunning = true;
+                callbackTarget = true;
+                runCallback();
+            }
+        }
+
+        /** [P] CommsCallback.stop (on a disconnect and on a lost connection): 
returns at once. */
+        void stopCallback() {
+            callbackTarget = false;
+            if (!busy) {
+                callbackRunning = false;
+            }
+        }
+
+        private void runCallback() {
+            if (!callbackRunning || workerScheduled || inbox.isEmpty()) {
+                return;
+            }
+            workerScheduled = true;
+            callbackThread.execute(() -> {
+                workerScheduled = false;
+                if (!callbackTarget) {
+                    callbackRunning = false;
+                    return;
+                }
+                if (inbox.isEmpty()) {
+                    return;
+                }
+                Object[] item = inbox.remove(0);
+                busy = true;
+                try {
+                    handle((Msg) item[0], (int) item[1], (boolean) item[2], 
(int) item[3]);
+                } finally {
+                    busy = false;
+                }
+                if (!callbackTarget) {
+                    callbackRunning = false;
+                    dst.trace("mqtt: the callback thread stops");
+                    return;
+                }
+                runCallback();
+            });
+        }
+
+        /** [P] CommsCallback.handleMessage: delivers the message, then sends 
the PUBACK unless the acks are manual. */
+        private void handle(Msg m, int id, boolean dup, int e) {
+            MqttMessage message = new MqttMessage(("m" + 
m.n).getBytes(StandardCharsets.UTF_8));
+            message.setQos(m.qos);
+            message.setId(id);
+            message.setDuplicate(dup);
+            onDelivered.accept(m.n);
+            try {
+                callback.messageArrived(TOPIC, message);
+            } catch (Exception ex) {
+                // [P] in v5, CommsCallback.deliverMessage catches and ignores 
an Exception from messageArrived (only an
+                // Error closes the connection); the consumer should not throw 
one, so the fake counts it
+                violations.add("messageArrived threw " + ex);
+            }
+            if (e == epoch && m.qos == 1 && !manualAcks) {
+                ack(id);
+            }
+        }
+
+        private void callbackTask(String what, Runnable r) {
+            callbackThread.execute(() -> {
+                if (callback != null || what.startsWith("deliver")) {
+                    r.run();
+                }
+            });
+        }
+
+        /** [P] the connection was lost: the callback is told, and the 
automatic reconnect starts. */
+        void connectionLost() {
+            connectedFlag = false;
+            if (callback != null) {
+                MqttCallback cb = callback;
+                callbackTask("disconnected", () -> cb.disconnected(new 
MqttDisconnectResponse(
+                        new 
MqttException(MqttClientException.REASON_CODE_CONNECTION_LOST))));
+            }
+            if (options != null && options.isAutomaticReconnect() && 
!disconnectedByUser && !closed) {
+                reconnecting = true;
+                reconnectCycle = true;
+                scheduleReconnect(1000);
+            }
+        }
+
+        private void scheduleReconnect(long delay) {
+            reconnectThread.schedule("automatic reconnect", delay, () -> {
+                if (!reconnectCycle || connectedFlag || closed || 
disconnectedByUser || connecting) {
+                    return;
+                }
+                connecting = true;
+                try {
+                    dst.block(latency(), "reconnect");
+                    if (connectedFlag || closed || disconnectedByUser) {
+                        return;
+                    }
+                    connectToBroker();
+                    connectedFlag = true;
+                    reconnectCycle = false;
+                    callbackTask("connectComplete", () -> 
callback.connectComplete(true, URI));
+                } catch (MqttException e) {
+                    long max = Math.max(1000, options.getMaxReconnectDelay());
+                    scheduleReconnect(Math.min(delay * 2, max));
+                } finally {
+                    connecting = false;
+                }
+            });
+        }
+
+        private IMqttToken token(boolean present) {
+            return (IMqttToken) 
Proxy.newProxyInstance(IMqttToken.class.getClassLoader(), new Class<?>[] { 
IMqttToken.class },
+                    (p, method, args) -> switch (method.getName()) {
+                        case "getSessionPresent" -> present;
+                        case "isComplete" -> true;
+                        case "toString" -> "FakeMqtt.token";
+                        case "hashCode" -> System.identityHashCode(p);
+                        case "equals" -> p == args[0];
+                        default -> {
+                            violations.add("not modelled: IMqttToken." + 
method.getName());
+                            throw new 
UnsupportedOperationException(method.getName());
+                        }
+                    });
+        }
+    }
+}
diff --git 
a/components/camel-paho-mqtt5/src/test/java/org/apache/camel/component/paho/mqtt5/dst/PahoMqtt5ConsumerDstTest.java
 
b/components/camel-paho-mqtt5/src/test/java/org/apache/camel/component/paho/mqtt5/dst/PahoMqtt5ConsumerDstTest.java
new file mode 100644
index 000000000000..08e3fbfb40fb
--- /dev/null
+++ 
b/components/camel-paho-mqtt5/src/test/java/org/apache/camel/component/paho/mqtt5/dst/PahoMqtt5ConsumerDstTest.java
@@ -0,0 +1,37 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.camel.component.paho.mqtt5.dst;
+
+import org.apache.camel.test.dst.Dst;
+import org.apache.camel.test.dst.DstInstrument;
+import org.apache.camel.test.dst.DstTest;
+
+/**
+ * Deterministic simulation test of the camel-paho-mqtt5 consumer against a 
fake MQTT 5 broker and Paho client, with
+ * connections closed by the broker, a broker that cannot be reached, 
connections lost during a subscribe, failed and
+ * slow processing, and route stops and starts: a message is acknowledged only 
after its exchange completed (or failed
+ * for good), the client drops no message, and once the faults cleared every 
message is routed. A failing seed is
+ * replayed with {@code -Ddst.seed=<seed>}; {@code -Ddst.seeds=5000} runs more 
seeds.
+ */
+@DstInstrument("org.apache.camel.component.paho.mqtt5")
+class PahoMqtt5ConsumerDstTest {
+
+    @DstTest(seeds = 1000)
+    void everyMessageIsRoutedAfterReconnects(Dst dst) throws Exception {
+        new PahoMqtt5World(dst).run();
+    }
+}
diff --git 
a/components/camel-paho-mqtt5/src/test/java/org/apache/camel/component/paho/mqtt5/dst/PahoMqtt5World.java
 
b/components/camel-paho-mqtt5/src/test/java/org/apache/camel/component/paho/mqtt5/dst/PahoMqtt5World.java
new file mode 100644
index 000000000000..cf475f533d37
--- /dev/null
+++ 
b/components/camel-paho-mqtt5/src/test/java/org/apache/camel/component/paho/mqtt5/dst/PahoMqtt5World.java
@@ -0,0 +1,480 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.camel.component.paho.mqtt5.dst;
+
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+import java.util.TreeSet;
+
+import org.apache.camel.Exchange;
+import org.apache.camel.ServiceStatus;
+import org.apache.camel.builder.RouteBuilder;
+import org.apache.camel.component.paho.mqtt5.PahoMqtt5Configuration;
+import org.apache.camel.component.paho.mqtt5.PahoMqtt5DstComponent;
+import org.apache.camel.impl.DefaultCamelContext;
+import org.apache.camel.test.dst.Dst;
+import org.apache.camel.test.dst.DstCamel;
+import org.apache.camel.test.dst.DstExecutor;
+import org.apache.camel.test.dst.DstWorld;
+import org.apache.camel.test.dst.Invariants;
+
+/**
+ * The world around the camel-paho-mqtt5 consumer: a fake MQTT 5 broker and 
the Paho client of the consumer
+ * ({@link FakeMqtt}), the client given to the real consumer through {@link 
PahoMqtt5DstComponent} (the consumer creates
+ * its own client in a package-private method), a publisher, a route whose 
processing is sometimes slow and sometimes
+ * fails, and an operator that drops the connection of the client, makes the 
broker unreachable for a while, and stops
+ * and starts the route (only while no message is in the route).
+ * <p>
+ * The seed picks the QoS, manualAcksEnabled, cleanStart, 
sessionExpiryInterval, automaticReconnect, receiveMaximum,
+ * maxReconnectDelay and the fault rates.
+ * <p>
+ * Properties: with manualAcksEnabled a QoS 1 message is acknowledged only 
after its exchange completed, or after it
+ * failed when the broker would not send it again; the client drops no message 
(a message that arrives before the
+ * callback is set); once the faults cleared every message that the broker 
queued for the session (and did not lose with
+ * a session that ended) is routed, a QoS 1 message with manual acks on a kept 
session completes, and nothing stays in
+ * flight or queued; after the final stop the client is disconnected and 
closed; the consumer uses the client as Paho
+ * allows.
+ * <p>
+ * The operator stops the route only while no message is in the route, and a 
message that arrives while a stop runs (the
+ * operator's, or the restart of the consumer after a failed subscribe) is 
processed at once: on the one stack the
+ * shutdown could not see an exchange that blocks below it finish. For the 
same reason a subscribe fails only while no
+ * message is in the route. Without automaticReconnect Paho does not reconnect 
after the broker closed the connection,
+ * so the operator restarts the route then. A message delivered while the 
route stops is refused by the route; without
+ * manual acknowledgements, or when the broker does not keep the session, its 
acknowledgement leaves it lost (the
+ * consumer's Javadoc of rollback describes it), and the world does not count 
it.
+ * <p>
+ * Two cases are looked at separately. A connect that comes while the callback 
thread of Paho is still in messageArrived
+ * after the connection closed (the reconnect of the consumer after a failed 
message, or the automatic reconnect, while
+ * another message is in the route) never completes, as read in the byte code 
of Paho 1.2.5 (not reproduced against a
+ * broker yet, and the fake leaves out the quiesce of a disconnect, which 
changes when the thread is busy): the client
+ * stays connecting, the consumer delivers nothing more, and the stop cannot 
close the client. This is CAMEL-27043 (it
+ * still needs a reproduction, and may be a Paho issue). When the client of 
the consumer hangs that way at the end of
+ * the run, the world does not check that every message is routed, and a hung 
client is not checked for being closed;
+ * {@code -Ddst.paho.strictCallbackThread=true} checks both. A stop while the 
client connects makes the close of the
+ * client fail (Paho throws 32110), and the consumer drops a client that goes 
on connecting, with its automatic
+ * reconnect and the same client id: CAMEL-27041. The operator does not stop 
the route then, and
+ * {@code -Ddst.paho.stopWhileConnecting=true} lets it.
+ * <p>
+ * Not explored: a client given with the client option (shared by several 
consumers), the persistence of the client,
+ * retained messages and wildcards, an acknowledgement that blocks, and a stop 
while a message is in the route.
+ */
+@DstWorld(component = "paho-mqtt5",
+          fake = "FakeMqtt: an MQTT 5 broker with the session of the client id 
(Clean Start, Session Expiry, Receive"
+                 + " Maximum, resend on reconnect) and a Paho MqttClient 
subclass on it, given to the consumer by a"
+                 + " test component",
+          covers = {
+                  "QoS 0, 1 and 2 with automatic or manual acknowledgements 
(manualAcksEnabled)",
+                  "a session that the broker keeps (cleanStart=false, 
sessionExpiryInterval) or not",
+                  "the automatic reconnect of Paho and the subscription made 
again",
+                  "the reconnect of the consumer after a failed message, and 
Receive Maximum",
+                  "route stop and start, with messages queued for a kept 
session" },
+          faults = {
+                  "the broker closes the connection", "the broker cannot be 
reached",
+                  "the connection is lost during a subscribe",
+                  "the broker answers slowly", "processing fails", "processing 
is slow" },
+          invariants = {
+                  "the consumer uses the client as Paho allows",
+                  "a message is acknowledged only after its exchange 
completed, or failed for good",
+                  "the client drops no message",
+                  "every message is routed once the faults cleared",
+                  "the stopped consumer disconnects and closes its client" },
+          issues = { "CAMEL-25793", "CAMEL-25944", "CAMEL-27041", 
"CAMEL-27043" })
+final class PahoMqtt5World {
+
+    static final String ROUTE = "mqtt";
+
+    private final Dst dst;
+    private final FakeMqtt mqtt;
+    private final int qos;
+    private final boolean manualAcks;
+    private final boolean cleanStart;
+    private final long sessionExpiry;
+    private final boolean automaticReconnect;
+    private final int receiveMaximum;
+    private final int maxReconnectDelay;
+    private final double processFailRate;
+    private final List<String> violations = new ArrayList<>();
+    private final Set<Long> published = new TreeSet<>();
+    private final Set<Long> routed = new HashSet<>();
+    private final Set<Long> completed = new HashSet<>();
+    private final Set<Long> failed = new HashSet<>();
+    private final Set<Long> delivered = new HashSet<>();
+    private final Set<Long> deliveredWhileStopping = new HashSet<>();
+    private DefaultCamelContext context;
+    private DstExecutor operator;
+    private boolean faults = true;
+    private int sendsLeft;
+    private int opsLeft;
+    private int inRoute;
+    private long nextMessage;
+    private boolean finished;
+    // whether the finisher made its check at the deadline
+    private boolean checked;
+    private boolean stopping;
+    private boolean restartByConsumer;
+    // CAMEL-27041: a route stopped while the client connects leaves that 
client connecting
+    private final boolean stopWhileConnecting = 
Boolean.getBoolean("dst.paho.stopWhileConnecting");
+    // CAMEL-27043: a connect while the callback thread is still in 
messageArrived hangs
+    private final boolean strictCallbackThread = 
Boolean.getBoolean("dst.paho.strictCallbackThread");
+    private String finalCheck;
+    private String afterStop;
+
+    PahoMqtt5World(Dst dst) {
+        this.dst = dst;
+        this.mqtt = new FakeMqtt(dst);
+        this.qos = dst.pick(List.of(0, 1, 1, 1, 2));
+        this.manualAcks = dst.chance(0.6);
+        this.cleanStart = dst.chance(0.4);
+        this.sessionExpiry = dst.pick(List.of(-1L, 60L, 3600L));
+        this.automaticReconnect = dst.chance(0.8);
+        this.receiveMaximum = dst.pick(List.of(1, 2, 10, 65535));
+        this.maxReconnectDelay = dst.pick(List.of(2000, 128000));
+        this.processFailRate = dst.pick(List.of(0.0, 0.1, 0.3));
+        mqtt.subscribeFailRate = dst.pick(List.of(0.0, 0.1));
+        mqtt.slowRate = dst.pick(List.of(0.0, 0.05));
+        mqtt.queueQos0 = dst.chance(0.5);
+        mqtt.onAck = this::acknowledged;
+        mqtt.onDelivered = this::delivered;
+        // a subscribe that fails when the client reconnected makes the 
consumer restart its route; on the one stack that
+        // shutdown could not see an exchange that blocks below it finish, so 
a subscribe fails only while no message is
+        // in the route
+        mqtt.subscribeFaultAllowed = () -> inRoute == 0 && !stopping;
+        // after a failed subscribe the consumer restarts its route (a 
reconnect) or fails to start: until a subscribe
+        // succeeds again the world treats the route as stopping
+        mqtt.onSubscribeFailed = () -> restartByConsumer = true;
+        mqtt.onSubscribed = () -> restartByConsumer = false;
+        mqtt.onDropped = n -> violations.add("the client dropped message " + n 
+ " at t=" + dst.now());
+        dst.trace("world: " + config());
+    }
+
+    private String config() {
+        return "qos=" + qos + " manualAcksEnabled=" + manualAcks + " 
cleanStart=" + cleanStart + " sessionExpiryInterval="
+               + sessionExpiry + " automaticReconnect=" + automaticReconnect + 
" receiveMaximum=" + receiveMaximum
+               + " maxReconnectDelay=" + maxReconnectDelay + " processFail=" + 
processFailRate + " subscribeFail="
+               + mqtt.subscribeFailRate + " slow=" + mqtt.slowRate + " 
queueQos0=" + mqtt.queueQos0;
+    }
+
+    /** Whether the broker keeps the session after the connection closes, so 
that it sends a QoS 1 message again. */
+    private boolean sessionKept() {
+        return !cleanStart && sessionExpiry > 0;
+    }
+
+    void run() throws Exception {
+        DefaultCamelContext context = DstCamel.newContext(dst);
+        this.context = context;
+        PahoMqtt5DstComponent component = new PahoMqtt5DstComponent(context, 
mqtt::newClient);
+        PahoMqtt5Configuration configuration = new PahoMqtt5Configuration();
+        configuration.setBrokerUrl(FakeMqtt.URI);
+        configuration.setClientId("dst-consumer");
+        configuration.setQos(qos);
+        configuration.setManualAcksEnabled(manualAcks);
+        configuration.setCleanStart(cleanStart);
+        configuration.setSessionExpiryInterval(sessionExpiry);
+        configuration.setAutomaticReconnect(automaticReconnect);
+        configuration.setReceiveMaximum(receiveMaximum);
+        configuration.setMaxReconnectDelay(maxReconnectDelay);
+        component.setConfiguration(configuration);
+        context.addComponent("paho-mqtt5", component);
+        String uri = "paho-mqtt5:" + FakeMqtt.TOPIC;
+        context.addRoutes(new RouteBuilder() {
+            @Override
+            public void configure() {
+                from(uri).routeId(ROUTE).autoStartup(false).process(exchange 
-> process(exchange));
+            }
+        });
+        context.start();
+        DstExecutor operator = dst.executor("operator", 1);
+        this.operator = operator;
+        try {
+            operator.schedule("start route", 0, () -> startRoute(context));
+            planSends();
+            planOperations(context, operator);
+            // the finisher runs on its own executor, so that it never waits 
behind an operation of the operator
+            DstExecutor finisher = dst.executor("finisher", 1);
+            finisher.schedule("finisher", 1000, () -> finisher(context, 
finisher, -1));
+            dst.runUntil(() -> finished, "the finisher");
+            dst.check(
+                    Invariants.of("the consumer uses the client as Paho 
allows",
+                            () -> mqtt.violations.isEmpty() ? null : 
mqtt.violations + " (" + config() + ")"),
+                    Invariants.of("a message is acknowledged only after its 
exchange completed, or failed for good",
+                            () -> violations.stream().noneMatch(v -> 
v.startsWith("acknowledged"))
+                                    ? null : violations + " (" + config() + 
")"),
+                    Invariants.of("the client drops no message",
+                            () -> violations.stream().noneMatch(v -> 
v.startsWith("the client dropped"))
+                                    ? null : violations + " (" + config() + 
")"),
+                    Invariants.of("every message is routed once the faults 
cleared",
+                            () -> finalCheck == null ? null : finalCheck + " 
(" + config() + ")"),
+                    Invariants.of("the stopped consumer disconnects and closes 
its client",
+                            () -> afterStop == null ? null : afterStop + " (" 
+ config() + ")"),
+                    Invariants.of("the world itself ran as planned",
+                            () -> violations.stream().noneMatch(v -> 
v.startsWith("world"))
+                                    ? null : violations + " (" + config() + 
")"));
+        } finally {
+            dst.dropPending();
+            context.stop();
+        }
+    }
+
+    private void process(Exchange exchange) {
+        String body = exchange.getMessage().getBody(String.class);
+        long n = Long.parseLong(body.substring(1));
+        inRoute++;
+        try {
+            routed.add(n);
+            if (stopping || restartByConsumer) {
+                // the route is being stopped: on the one stack the shutdown 
could not see an exchange that blocks below
+                // it finish, so an exchange that starts now takes no time
+                dst.trace("world: message " + n + " processed at once, the 
route is stopping");
+            } else if (faults && dst.chance(0.05)) {
+                dst.block(dst.between(1000, 10_000), "slow processing of " + 
n);
+            } else {
+                dst.block(dst.between(0, 100), "processing of " + n);
+            }
+            if (faults && dst.chance(processFailRate)) {
+                failed.add(n);
+                dst.trace("world: message " + n + " failed");
+                throw Dst.injected(new IllegalStateException("dst: processing 
failed for message " + n));
+            }
+            completed.add(n);
+            dst.trace("world: message " + n + " completed");
+        } finally {
+            inRoute--;
+        }
+    }
+
+    private void delivered(long n) {
+        delivered.add(n);
+        ServiceStatus status = context != null ? 
context.getRouteController().getRouteStatus(ROUTE) : null;
+        if (stopping || restartByConsumer || status == ServiceStatus.Stopping 
|| status == ServiceStatus.Stopped) {
+            // the route stops: the exchange of this message fails, and 
without manual acks (or on a session the broker
+            // does not keep) its acknowledgement leaves it lost. Not while 
the route starts: a kept session sends its
+            // messages right after the connect, and the consumer must route 
them
+            deliveredWhileStopping.add(n);
+        }
+    }
+
+    private void acknowledged(long n) {
+        if (!manualAcks || qos != 1) {
+            return;
+        }
+        if (completed.contains(n)) {
+            return;
+        }
+        if (!sessionKept() && (failed.contains(n) || 
deliveredWhileStopping.contains(n) && !routed.contains(n))) {
+            // the exchange failed (or was refused by a route that stops), and 
the broker would never send the message
+            // again: the consumer acknowledges it
+            return;
+        }
+        violations.add("acknowledged message " + n + " before its exchange 
completed (failed=" + failed.contains(n)
+                       + ") at t=" + dst.now());
+    }
+
+    // 
------------------------------------------------------------------------------------------------------------
+    // the load and the operator
+
+    private void planSends() {
+        DstExecutor publisher = dst.executor("publisher", 1);
+        long t = dst.between(200, 1000);
+        int bursts = (int) dst.between(1, 4);
+        for (int b = 0; b < bursts; b++) {
+            int size = (int) dst.between(1, 15);
+            for (int i = 0; i < size; i++) {
+                long n = nextMessage++;
+                sendsLeft++;
+                publisher.schedule("publish " + n, t, () -> {
+                    if (mqtt.isSubscribed()) {
+                        published.add(n);
+                    }
+                    mqtt.publish(n, qos);
+                    sendsLeft--;
+                });
+                t += dst.between(0, 300);
+            }
+            t += dst.between(0, 20_000);
+        }
+    }
+
+    private void planOperations(DefaultCamelContext context, DstExecutor 
operator) {
+        int ops = (int) dst.between(0, 4);
+        long t = dst.between(1000, 5000);
+        for (int i = 0; i < ops; i++) {
+            opsLeft++;
+            int kind = (int) dst.between(0, 3);
+            if (kind == 0) {
+                operator.schedule("drop connection", t, () -> {
+                    mqtt.dropConnection();
+                    opsLeft--;
+                });
+            } else if (kind == 1) {
+                long length = dst.between(500, 20_000);
+                operator.schedule("broker unreachable", t, () -> {
+                    mqtt.unreachable(length);
+                    opsLeft--;
+                });
+            } else {
+                operator.schedule("stop route", t, () -> stopWhenIdle(context, 
operator));
+            }
+            t += dst.between(1000, 20_000);
+        }
+    }
+
+    private void startRoute(DefaultCamelContext context) {
+        try {
+            context.getRouteController().startRoute(ROUTE);
+        } catch (Exception e) {
+            // the broker cannot be reached: the route stays stopped, and the 
operator starts it again later
+            dst.trace("world: route start failed: " + e.getMessage());
+            operator.schedule("start route again", 2000, () -> 
startRoute(context));
+        }
+    }
+
+    /** Stops the route once no message is in it, then starts it again after a 
while. */
+    private void stopWhenIdle(DefaultCamelContext context, DstExecutor 
operator) {
+        if (context.getRouteController().getRouteStatus(ROUTE) != 
ServiceStatus.Started) {
+            opsLeft--;
+            return;
+        }
+        if (mqtt.anyHanging()) {
+            // a hung connect keeps the message below it on the one stack in 
the route until the end of the run: no stop
+            dst.trace("world: no route stop, as a connect hangs");
+            opsLeft--;
+            return;
+        }
+        if (inRoute > 0 || !stopWhileConnecting && mqtt.anyConnecting()) {
+            operator.schedule("stop route (later)", 50, () -> 
stopWhenIdle(context, operator));
+            return;
+        }
+        stopRoute(context);
+        operator.schedule("start route", dst.between(100, 10_000), () -> {
+            startRoute(context);
+            opsLeft--;
+        });
+    }
+
+    private void restart(DefaultCamelContext context) {
+        dst.trace("world: restart the route, as the client is disconnected and 
does not reconnect");
+        stopRoute(context);
+        startRoute(context);
+    }
+
+    private void stopRoute(DefaultCamelContext context) {
+        stopping = true;
+        try {
+            context.getRouteController().stopRoute(ROUTE);
+        } catch (Exception e) {
+            violations.add("world: stop failed: " + e);
+        } finally {
+            stopping = false;
+        }
+    }
+
+    private Set<Long> missing() {
+        Set<Long> missing = new TreeSet<>();
+        for (long n : published) {
+            if (mqtt.lostWithSession.contains(n) || 
mqtt.lostInClient.contains(n)) {
+                continue;
+            }
+            if (deliveredWhileStopping.contains(n) && !routed.contains(n) && 
(qos != 1 || !manualAcks || !sessionKept())) {
+                // delivered while the route stopped, refused and acknowledged 
(see delivered)
+                continue;
+            }
+            if (qos == 0 && !delivered.contains(n)) {
+                // at most once: a QoS 0 message that the client never got is 
lost
+                continue;
+            }
+            boolean mustComplete = qos == 1 && manualAcks && sessionKept();
+            if (mustComplete ? !completed.contains(n) : !routed.contains(n)) {
+                missing.add(n);
+            }
+        }
+        return missing;
+    }
+
+    private void finisher(DefaultCamelContext context, DstExecutor self, long 
deadline) {
+        if (deadline == -1 && (sendsLeft > 0 || opsLeft > 0)) {
+            self.schedule("finisher", 500, () -> finisher(context, self, -1));
+            return;
+        }
+        if (deadline == -1) {
+            faults = false;
+            mqtt.faults = false;
+            dst.trace("world: faults cleared");
+            long d = dst.now() + 10 * 60_000L;
+            self.schedule("finisher", 500, () -> finisher(context, self, d));
+            return;
+        }
+        ServiceStatus status = 
context.getRouteController().getRouteStatus(ROUTE);
+        Set<Long> missing = missing();
+        boolean settled = status == ServiceStatus.Started && 
mqtt.isConnected() && missing.isEmpty() && inRoute == 0
+                && (qos == 0 || mqtt.pending() == 0);
+        if (!settled && dst.now() < deadline) {
+            if (status == ServiceStatus.Stopped && dst.now() > deadline - 9 * 
60_000L) {
+                // a route that stopped on its own (a failed start) is started 
again
+                startRoute(context);
+            } else if (status == ServiceStatus.Started && !automaticReconnect 
&& !mqtt.isConnected() && inRoute == 0
+                    && !mqtt.anyConnecting()) {
+                // without automaticReconnect Paho does not reconnect after 
the broker closed the
+                // connection: the operator restarts the route
+                restart(context);
+            }
+            self.schedule("finisher", 500, () -> finisher(context, self, 
deadline));
+            return;
+        }
+        boolean excuseHung = !strictCallbackThread;
+        if (!checked) {
+            checked = true;
+            if (!settled && excuseHung && mqtt.currentHung()) {
+                // the client hangs in a connect (see the class Javadoc): 
counted, its messages are not checked
+                dst.trace("world: the client hangs in a connect (" + 
mqtt.hungConnects
+                          + "); the messages are not checked");
+            } else if (!settled) {
+                finalCheck = "at the deadline: route " + status + ", 
connected=" + mqtt.isConnected() + ", subscribed="
+                             + mqtt.isSubscribed() + ", never routed (or not 
completed) " + missing
+                             + ", in flight or queued at the broker " + 
mqtt.pending();
+            }
+            // the hung connects end here (they fail), so that the messages 
below them on the one stack can complete
+            // and the run can end; Paho would wait for ever
+            mqtt.releaseHung = true;
+        }
+        if (inRoute > 0 || !stopWhileConnecting && mqtt.anyConnecting()) {
+            self.schedule("finisher (later)", 50, () -> finisher(context, 
self, deadline));
+            return;
+        }
+        stopRoute(context);
+        List<String> open = new ArrayList<>();
+        for (FakeMqtt.Client c : mqtt.clients()) {
+            if (excuseHung && c.isHung()) {
+                // a hung client cannot be closed (32110), see the class 
Javadoc
+                continue;
+            }
+            if (c.isConnected()) {
+                open.add("a client is still connected");
+            } else if (!c.isClosed()) {
+                open.add("a client is not closed");
+            }
+        }
+        if (mqtt.isConnected()) {
+            open.add("the broker still has a connection");
+        }
+        if (!open.isEmpty()) {
+            afterStop = String.join(", ", open);
+        }
+        finished = true;
+    }
+}
diff --git a/components/camel-paho-mqtt5/src/test/resources/log4j2.properties 
b/components/camel-paho-mqtt5/src/test/resources/log4j2.properties
index 49635d53ff7e..d51928b0a6cc 100644
--- a/components/camel-paho-mqtt5/src/test/resources/log4j2.properties
+++ b/components/camel-paho-mqtt5/src/test/resources/log4j2.properties
@@ -15,9 +15,16 @@
 ## limitations under the License.
 ## ---------------------------------------------------------------------------
 
-appender.file.type = File
+# capped: the DST test runs the consumer for a thousand seeds
+appender.file.type = RollingFile
 appender.file.name = file
 appender.file.fileName = target/camel-paho-mqtt5-test.log
+appender.file.filePattern = target/camel-paho-mqtt5-test.%i.log
+appender.file.policies.type = Policies
+appender.file.policies.size.type = SizeBasedTriggeringPolicy
+appender.file.policies.size.size = 50MB
+appender.file.strategy.type = DefaultRolloverStrategy
+appender.file.strategy.max = 1
 appender.file.layout.type = PatternLayout
 appender.file.layout.pattern = %d [%-15.15t] %-5p %-30.30c{1} - %m%n
 appender.out.type = Console
diff --git a/components/camel-test/camel-test-dst/src/main/docs/test-dst.adoc 
b/components/camel-test/camel-test-dst/src/main/docs/test-dst.adoc
index 4923ed2b2935..43249aaa404e 100644
--- a/components/camel-test/camel-test-dst/src/main/docs/test-dst.adoc
+++ b/components/camel-test/camel-test-dst/src/main/docs/test-dst.adoc
@@ -134,6 +134,16 @@ a|* the JetStream pull consumer with explicit acks and 
naks, ephemeral or durabl
 * a server restart that deletes the ephemeral consumer, and the subscription 
made again
 |CAMEL-26884
 
+|camel-paho-mqtt5
+|`PahoMqtt5World` (`PahoMqtt5ConsumerDstTest`)
+|FakeMqtt: an MQTT 5 broker with the session of the client id (Clean Start, 
Session Expiry, Receive Maximum, resend on reconnect) and a Paho MqttClient 
subclass on it, given to the consumer by a test component
+a|* QoS 0, 1 and 2 with automatic or manual acknowledgements 
(manualAcksEnabled)
+* a session that the broker keeps (cleanStart=false, sessionExpiryInterval) or 
not
+* the automatic reconnect of Paho and the subscription made again
+* the reconnect of the consumer after a failed message, and Receive Maximum
+* route stop and start, with messages queued for a kept session
+|CAMEL-25793, CAMEL-25944, CAMEL-27041, CAMEL-27043
+
 |camel-pulsar
 |`PulsarWorld` (`PulsarConsumerDstTest`)
 |FakePulsar: a broker and client for one topic and one durable subscription
@@ -615,6 +625,12 @@ CAMEL-25737), `dst.replayCheckSeed` (the seed of the 
in-JVM replay test of the E
 `dst.share.pollOnError` (the pollOnError strategies the Kafka share world 
picks from)
 and `dst.pulsar.subscribeFailRates` (the subscribe failure rates of the Pulsar 
world, `0` runs without subscribe failures).
 
+For camel-paho-mqtt5 (`components/camel-paho-mqtt5`, 
`PahoMqtt5ConsumerDstTest`) the past fixes are the main code of
+6044f38166f9 (CAMEL-25944) and the change of b7594f06e0b7 (CAMEL-25793), taken 
out by hand. Its world has two
+properties, each adding back a case that the default set leaves out: 
`dst.paho.strictCallbackThread` (`true` checks a
+client whose connect hangs while the callback thread is still in 
messageArrived, CAMEL-27043) and
+`dst.paho.stopWhileConnecting` (`true` lets the operator stop the route while 
the client connects, CAMEL-27041).
+
 == Limits
 
 * Interleaving is per task: the harness does not preempt a task in the middle 
of a method.

Reply via email to