sharanggupta opened a new issue, #2732: URL: https://github.com/apache/activemq/issues/2732
**Component:** activemq-client (failover, connection audit) **Version:** present on `main` as of 2026-10-21, commit `c5f98aedc` ### Description `FailoverDurableSubTransactionTest` has been `@Ignore`d since #1621, waiting for AMQ-9829 (#1616). That change is merged, but with the `@Ignore` removed the class still fails about 6 of 21 runs locally (JDK 17, macOS). There are two separate client-side races. In both, a message that the broker legitimately replays after failover is poison-acked as a duplicate and ends up in the DLQ. #### 1. Listener path: `testFailoverCommitListener`, DLQ count of 1 to 10 1. Messages dispatched before `setMessageListener` are recorded by the connection audit ([`ActiveMQMessageConsumer.java:1662`](https://github.com/apache/activemq/blob/c5f98aedc/activemq-client/src/main/java/org/apache/activemq/ActiveMQMessageConsumer.java#L1662)) and parked in the consumer's buffer with `deliverySequenceId = -1` ([`:1689`](https://github.com/apache/activemq/blob/c5f98aedc/activemq-client/src/main/java/org/apache/activemq/ActiveMQMessageConsumer.java#L1689)). 2. `setMessageListener` moves them into the session executor queue through [`ActiveMQSession.redispatch`](https://github.com/apache/activemq/blob/c5f98aedc/activemq-client/src/main/java/org/apache/activemq/ActiveMQSession.java#L2189). 3. On a transport interruption, [`ActiveMQSessionExecutor.clearMessagesInProgress()`](https://github.com/apache/activemq/blob/c5f98aedc/activemq-client/src/main/java/org/apache/activemq/ActiveMQSessionExecutor.java#L179-L181) clears that queue but leaves the messages in the audit. The AMQ-9829 tracking only covers the consumer's own buffer, which is empty at that point. 4. When the broker replays them after the reconnect, the client logs "suppressing duplicate delivery on connection, poison acking" ([`ActiveMQMessageConsumer.java:1726`](https://github.com/apache/activemq/blob/c5f98aedc/activemq-client/src/main/java/org/apache/activemq/ActiveMQMessageConsumer.java#L1726)) and the messages go to the DLQ. The DLQ count equals the number of messages that were prefetched before the listener was set. If the test waits for the full prefetch before calling `setMessageListener`, the ON_ACK and ON_COMMIT variants put 10 messages in the DLQ on every run. #### 2. `receive()` path: `testFailoverCommit`, DLQ count of 1, or "COMMIT FAILED: Transaction marked rollback only" 1. `receive()` dequeues a dispatch ([`:506`](https://github.com/apache/activemq/blob/c5f98aedc/activemq-client/src/main/java/org/apache/activemq/ActiveMQMessageConsumer.java#L506)) and only later adds it to `deliveredMessages` ([`:1163`](https://github.com/apache/activemq/blob/c5f98aedc/activemq-client/src/main/java/org/apache/activemq/ActiveMQMessageConsumer.java#L1163)). 2. If `clearMessagesInProgress()` and `clearDeliveredList()` run inside that window, the message is neither tracked as prefetched or delivered nor rolled back in the audit, so its replay is poison-acked. 3. A temporary 20 ms sleep inside that window makes three of the four `testFailoverCommit` variants fail on every run. With ON_DISPATCH, when the message is the first one in the transaction, the later ack is unmatched, the broker marks the transaction rollback-only, and `commit()` throws a plain `JMSException` that the test does not expect. ### Proposed fix I have a branch for the first race: [`sharanggupta:amq-9829-failover-durable-sub-race`](https://github.com/sharanggupta/activemq/compare/main...sharanggupta:activemq:amq-9829-failover-durable-sub-race). - `ActiveMQSessionExecutor.clearMessagesInProgress()` rolls back the audit entry for every dropped dispatch that already passed the duplicate check (non-zero `deliverySequenceId`), as the consumer already does for its own buffer. Browser consumers are skipped, as in the consumer. - It does not also record the dropped messages as prefetched (`capturePrefetchedMessagesForDuplicateSuppression`), as the consumer does for transacted sessions. That map is guarded by the consumer's locks, and this code runs on the transport thread, which the comment in `ActiveMQSession.clearMessagesInProgress` says must not take them. Rolling back the audit entry is enough for the replay to be accepted. If you would rather keep the two paths identical, the session could hand the dropped dispatches to the consumer's deferred clear instead. - `testFailoverCommitListener` is re-enabled and waits for the full prefetch before setting the listener. Without the fix it fails on every run with 10 messages in the DLQ. With the fix it passed 25 of 25 runs. - `testFailoverCommit` stays ignored, with a note pointing at the second race. Fixing that one probably means making the in-flight dequeued dispatch visible to the clear logic, which is a larger change. The session executor has the same kind of window: a dispatch that `iterate()` has taken off its queue but not yet handed to the consumer is missed by the clear. I can open a pull request for the first part if this direction looks right. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected] For further information, visit: https://activemq.apache.org/contact
