We have been experiencing issues with duplicate issues in several of our
production environments. The behaviour we see is that sometimes when our
application starts it logs a “suppressing duplicate delivery” message for a
message that has never been acknowledged. After this occurs the consumers do
not consume any more messages until we restart the application. Normally that
resolves the issue, the message is consumed successfully and continues to
consume messages. Occasionally it gets into the same state so we restart again.
An example of the suppressing duplicate delivery messages is;
ID:ip-10-210-179-152-19241-1722563936693-1:6:5:1 suppressing duplicate delivery
on connection, poison acking: MessageDispatch {commandId = 0, responseRequired
= false, consumerId = ID:ip-10-210-179-152-19241-1722563936693-1:6:5:1,
destination =queue://TEMPLATE_ORCHESTRATION_MESSAGES, message =
ActiveMQObjectMessage {commandId = 27442, responseRequired = true, messageId =
ID:ip-10-210-181-64-5463-1722503120609-7:1:50:1:980, originalDestination =
null, originalTransactionId = null, producerId
=ID:ip-10-210-181-64-5463-1722503120609-7:1:50:1, destination =
queue://TEMPLATE_ORCHESTRATION_MESSAGES, transactionId = null, deliveryTime =
0, expiration = 0, timestamp = 1722565114517, arrival = 0, brokerInTime =
1722565319026, brokerOutTime = 1722565319026, correlationId = null, replyTo =
null, persistent = true, type = null, priority = 4, groupID = null,
groupSequence = 0, targetConsumerId = null, compressed = false, userID = null,
content =
org.apache.activemq.util.ByteSequence@222083bc<mailto:org.apache.activemq.util.ByteSequence@222083bc>,
marshalledProperties =
org.apache.activemq.util.ByteSequence@44247f43<mailto:org.apache.activemq.util.ByteSequence@44247f43>,
dataStructure = null, redeliveryCounter = 3, size = 0, properties =
{WORKSPACE_ID=66ac0e4ef78b094c390c1d6b,
x__dash__datadog__dash__trace__dash__id=8195324751683450459,x__dash__datadog__dash__sampling__dash__priority=1,
x__dash__datadog__dash__parent__dash__id=2836570378207542649,scheduledJobId=ID:ip-10-210-181-64-5463-1722503120609-7:1:50:1:529018,
TEMPLATE_PURCHASE_ID=66ac0e4ef78b094c390c1d7b,tracestate=dd=t.dm:-1;t.tid:66ac41bd00000000,
CALLER_REFERENCE_ID=66ac0e470cfa9827bc10744c,
traceparent=00-66ac41bd0000000071bba470e30ef65b-275d85baed74bd79-01,
redeliveryDelay=25000,x__dash__datadog__dash__tags=_dd.p.dm=-1,_dd.p.tid=66ac41bd00000000},
readOnlyProperties = true, readOnlyBody = true, droppable = false,
jmsXGroupFirstForConsumer = false}, redeliveryCounter = 3}
We are using Amazon MQ with ActiveMQ version 5.16.3 (although we experienced
this issue on previous versions too).
Our java application is using org.apache.activemq:activemq-client:5.16.3.
The Spring config is;
@Bean
public ConnectionFactory
jmsConsumerConnectionFactory(@Value("${jms.broker.url}") String jmsBrokerUrl,
@Value("${jms.broker.username}") String username,
@Value("${jms.broker.password}") String password,
RedeliveryPolicyMap
consumerRedeliveryPolicyMap,
ActiveMQPrefetchPolicy
prefetchPolicy) {
return activeMQConnectionFactory(jmsBrokerUrl, username, password,
consumerRedeliveryPolicyMap, prefetchPolicy, "enc-consumer");
}
private ActiveMQConnectionFactory activeMQConnectionFactory(String
jmsBrokerUrl, String username, String password, RedeliveryPolicyMap
consumerRedeliveryPolicyMap, ActiveMQPrefetchPolicy prefetchPolicy, String
clientIdPrefix) {
ActiveMQConnectionFactory connectionFactory = new
ActiveMQConnectionFactory(jmsBrokerUrl);
connectionFactory.setUserName(trimToNull(username));
connectionFactory.setPassword(trimToNull(password));
connectionFactory.setObjectMessageSerializationDefered(true);
connectionFactory.setAlwaysSyncSend(true);
connectionFactory.setTrustAllPackages(true);
connectionFactory.setRedeliveryPolicyMap(consumerRedeliveryPolicyMap);
connectionFactory.setPrefetchPolicy(prefetchPolicy);
connectionFactory.setClientIDPrefix(clientIdPrefix);
return connectionFactory;
}
protected DefaultJmsListenerContainerFactory
defaultJmsListenerContainerFactory(boolean autoStartupEnabled, String
concurrency, ErrorHandler errorHandler) {
DefaultJmsListenerContainerFactory factory = new
DefaultJmsListenerContainerFactory();
factory.setConcurrency(concurrency);
factory.setConnectionFactory(jmsConsumerConnectionFactory);
factory.setTransactionManager(localJmsTransactionManager);
factory.setErrorHandler(errorHandler);
factory.setAutoStartup(autoStartupEnabled);
factory.setCacheLevel(CACHE_CONSUMER);
return factory;
}
and the broker URI is typically;
jms.broker.url="failover:(ssl://b-7cef9e05-66d6cbcf570b-1.mq.eu-west-2.amazonaws.com:61617,ssl://b-7cef9e05-66d6cbcf570b-2.mq.eu-west-2.amazonaws.com:61617)?initialReconnectDelay=100&maxReconnectAttempts=10&jms.checkForDuplicates=true&jms.watchTopicAdvisories=false&jms.prefetchPolicy.queuePrefetch=0&nested.wireFormat.maxInactivityDuration=1800000&nested.wireFormat.maxFrameSize=104857600"
I’ve debugged the client code to try and see where it’s detecting the duplicate
and it seems to have the duplicate message id in it’s cache. It’s difficult to
reproduce and so it’s hard to tell if does indeed get the same message twice
but given it doesn’t handle the message at all then I’m suspecting the cache is
incorrectly preloaded. Could it be that it’s caching to the filesystem and
when the application restarts it reloads the cache?
Would appreciate any help as we really want duplicate detection enabled.
Regards
Ben
[https://6032599.fs1.hubspotusercontent-na1.net/hubfs/6032599/Email%20images/Encompass%20logo/Encompass-(full)_300x52.png]
The leader in Corporate Digital Identity
Website<https://www.encompasscorporation.com/> |
LinkedIn<https://www.linkedin.com/company/encompass-corporation/>
Encompass Corporation UK Ltd | Company No. SC493055 | Address: Level 3, 33
Bothwell Street, Glasgow, UK, G2 6NL
Encompass Corporation Australia PTY Ltd | 605 414 499| Address: Level 1, 117
Clarence Street, Sydney, New South Wales, 2000
Encompass Corporation US Inc | Company No. 7946259 | Address: 11 West 42nd
Street, New York, New York, 10036
This email and any attachments is intended only for the use of the individual
or entity named above and may contain confidential information. If you are not
the intended recipient, any dissemination, distribution or copying of this
email is prohibited. If received in error, please notify us immediately by
return email and destroy the original message.
Public
