As far as I can tell, there is no logic in the OpenWire JMS client to
persist cache data to or load cache data from disk for duplicate detection
on the consumer. If the ID already exists in the cache then it appears that
it was added since the consumer was started. Also, it appears that entries
are added to the cache whether the message is acknowledged or not so the
fact that the message was not acknowledged doesn't to be relevant.
It's worth noting that 5.16.x is no longer active (and neither is 5.17.x)
so if any issue was found it would presumably be addressed on 5.18.x. Also,
it's possible that this issue (assuming one exists) has already been
addressed in the 3 years since 5.16.3 was released. Therefore, it might be
worth testing on the latest 5.18.x release.
Justin
On Sun, Aug 4, 2024 at 8:22 PM Ben D'Herville <b...@encompasscorporation.com>
wrote:
> We have been experiencing issues with duplicate issues in several of our
> production environments. The behaviour we see is that sometimes when our
> application starts it logs a “suppressing duplicate delivery” message for a
> message that has never been acknowledged. After this occurs the consumers
> do not consume any more messages until we restart the application.
> Normally that resolves the issue, the message is consumed successfully and
> continues to consume messages. Occasionally it gets into the same state so
> we restart again.
>
> An example of the suppressing duplicate delivery messages is;
> ID:ip-10-210-179-152-19241-1722563936693-1:6:5:1 suppressing duplicate
> delivery on connection, poison acking: MessageDispatch {commandId = 0,
> responseRequired = false, consumerId =
> ID:ip-10-210-179-152-19241-1722563936693-1:6:5:1, destination
> =queue://TEMPLATE_ORCHESTRATION_MESSAGES, message = ActiveMQObjectMessage
> {commandId = 27442, responseRequired = true, messageId =
> ID:ip-10-210-181-64-5463-1722503120609-7:1:50:1:980, originalDestination =
> null, originalTransactionId = null, producerId
> =ID:ip-10-210-181-64-5463-1722503120609-7:1:50:1, destination =
> queue://TEMPLATE_ORCHESTRATION_MESSAGES, transactionId = null, deliveryTime
> = 0, expiration = 0, timestamp = 1722565114517, arrival = 0, brokerInTime =
> 1722565319026, brokerOutTime = 1722565319026, correlationId = null, replyTo
> = null, persistent = true, type = null, priority = 4, groupID = null,
> groupSequence = 0, targetConsumerId = null, compressed = false, userID =
> null, content = org.apache.activemq.util.ByteSequence@222083bc<mailto:
> org.apache.activemq.util.ByteSequence@222083bc>, marshalledProperties =
> org.apache.activemq.util.ByteSequence@44247f43<mailto:
> org.apache.activemq.util.ByteSequence@44247f43>, dataStructure = null,
> redeliveryCounter = 3, size = 0, properties =
> {WORKSPACE_ID=66ac0e4ef78b094c390c1d6b,
> x__dash__datadog__dash__trace__dash__id=8195324751683450459,x__dash__datadog__dash__sampling__dash__priority=1,
> x__dash__datadog__dash__parent__dash__id=2836570378207542649,scheduledJobId=ID:ip-10-210-181-64-5463-1722503120609-7:1:50:1:529018,
> TEMPLATE_PURCHASE_ID=66ac0e4ef78b094c390c1d7b,tracestate=dd=t.dm:-1;t.tid:66ac41bd00000000,
> CALLER_REFERENCE_ID=66ac0e470cfa9827bc10744c,
> traceparent=00-66ac41bd0000000071bba470e30ef65b-275d85baed74bd79-01,
> redeliveryDelay=25000,x__dash__datadog__dash__tags=_dd.p.dm=-1,_dd.p.tid=66ac41bd00000000},
> readOnlyProperties = true, readOnlyBody = true, droppable = false,
> jmsXGroupFirstForConsumer = false}, redeliveryCounter = 3}
>
> We are using Amazon MQ with ActiveMQ version 5.16.3 (although we
> experienced this issue on previous versions too).
> Our java application is using org.apache.activemq:activemq-client:5.16.3.
>
> The Spring config is;
>
> @Bean
> public ConnectionFactory
> jmsConsumerConnectionFactory(@Value("${jms.broker.url}") String
> jmsBrokerUrl,
>
> @Value("${jms.broker.username}") String username,
>
> @Value("${jms.broker.password}") String password,
> RedeliveryPolicyMap
> consumerRedeliveryPolicyMap,
>
> ActiveMQPrefetchPolicy prefetchPolicy) {
> return activeMQConnectionFactory(jmsBrokerUrl, username, password,
> consumerRedeliveryPolicyMap, prefetchPolicy, "enc-consumer");
> }
>
> private ActiveMQConnectionFactory activeMQConnectionFactory(String
> jmsBrokerUrl, String username, String password, RedeliveryPolicyMap
> consumerRedeliveryPolicyMap, ActiveMQPrefetchPolicy prefetchPolicy, String
> clientIdPrefix) {
> ActiveMQConnectionFactory connectionFactory = new
> ActiveMQConnectionFactory(jmsBrokerUrl);
> connectionFactory.setUserName(trimToNull(username));
> connectionFactory.setPassword(trimToNull(password));
>
> connectionFactory.setObjectMessageSerializationDefered(true);
> connectionFactory.setAlwaysSyncSend(true);
> connectionFactory.setTrustAllPackages(true);
> connectionFactory.setRedeliveryPolicyMap(consumerRedeliveryPolicyMap);
> connectionFactory.setPrefetchPolicy(prefetchPolicy);
>
> connectionFactory.setClientIDPrefix(clientIdPrefix);
>
> return connectionFactory;
> }
>
>
> protected DefaultJmsListenerContainerFactory
> defaultJmsListenerContainerFactory(boolean autoStartupEnabled, String
> concurrency, ErrorHandler errorHandler) {
> DefaultJmsListenerContainerFactory factory = new
> DefaultJmsListenerContainerFactory();
> factory.setConcurrency(concurrency);
> factory.setConnectionFactory(jmsConsumerConnectionFactory);
> factory.setTransactionManager(localJmsTransactionManager);
> factory.setErrorHandler(errorHandler);
> factory.setAutoStartup(autoStartupEnabled);
> factory.setCacheLevel(CACHE_CONSUMER);
>
> return factory;
> }
>
> and the broker URI is typically;
>
>
> jms.broker.url="failover:(ssl://
> b-7cef9e05-66d6cbcf570b-1.mq.eu-west-2.amazonaws.com:61617
> ,ssl://b-7cef9e05-66d6cbcf570b-2.mq.eu-west-2.amazonaws.com:61617
> )?initialReconnectDelay=100&maxReconnectAttempts=10&jms.checkForDuplicates=true&jms.watchTopicAdvisories=false&jms.prefetchPolicy.queuePrefetch=0&nested.wireFormat.maxInactivityDuration=1800000&nested.wireFormat.maxFrameSize=104857600"
>
> I’ve debugged the client code to try and see where it’s detecting the
> duplicate and it seems to have the duplicate message id in it’s cache.
> It’s difficult to reproduce and so it’s hard to tell if does indeed get the
> same message twice but given it doesn’t handle the message at all then I’m
> suspecting the cache is incorrectly preloaded. Could it be that it’s
> caching to the filesystem and when the application restarts it reloads the
> cache?
>
> Would appreciate any help as we really want duplicate detection enabled.
>
> Regards
> Ben
>
>
>
>
> [
> https://6032599.fs1.hubspotusercontent-na1.net/hubfs/6032599/Email%20images/Encompass%20logo/Encompass-(full)_300x52.png
> ]
>
> The leader in Corporate Digital Identity
>
> Website<https://www.encompasscorporation.com/> | LinkedIn<
> https://www.linkedin.com/company/encompass-corporation/>
>
> Encompass Corporation UK Ltd | Company No. SC493055 | Address: Level 3, 33
> Bothwell Street, Glasgow, UK, G2 6NL
> Encompass Corporation Australia PTY Ltd | 605 414 499| Address: Level 1,
> 117 Clarence Street, Sydney, New South Wales, 2000
> Encompass Corporation US Inc | Company No. 7946259 | Address: 11 West 42nd
> Street, New York, New York, 10036
>
> This email and any attachments is intended only for the use of the
> individual or entity named above and may contain confidential information.
> If you are not the intended recipient, any dissemination, distribution or
> copying of this email is prohibited. If received in error, please notify us
> immediately by return email and destroy the original message.
>
>
>
>
>
>
> Public
>
