Hi Ben-
I suggest you open a ticket with AWS. The Apache community does not have
visibility to their build to know what patches or changes have been applied to
their build.
Thanks,
Matt Pavlovich
> On Aug 4, 2024, at 8:21 PM, Ben D'Herville <b...@encompasscorporation.com>
> wrote:
>
> We have been experiencing issues with duplicate issues in several of our
> production environments. The behaviour we see is that sometimes when our
> application starts it logs a “suppressing duplicate delivery” message for a
> message that has never been acknowledged. After this occurs the consumers do
> not consume any more messages until we restart the application. Normally
> that resolves the issue, the message is consumed successfully and continues
> to consume messages. Occasionally it gets into the same state so we restart
> again.
>
> An example of the suppressing duplicate delivery messages is;
> ID:ip-10-210-179-152-19241-1722563936693-1:6:5:1 suppressing duplicate
> delivery on connection, poison acking: MessageDispatch {commandId = 0,
> responseRequired = false, consumerId =
> ID:ip-10-210-179-152-19241-1722563936693-1:6:5:1, destination
> =queue://TEMPLATE_ORCHESTRATION_MESSAGES, message = ActiveMQObjectMessage
> {commandId = 27442, responseRequired = true, messageId =
> ID:ip-10-210-181-64-5463-1722503120609-7:1:50:1:980, originalDestination =
> null, originalTransactionId = null, producerId
> =ID:ip-10-210-181-64-5463-1722503120609-7:1:50:1, destination =
> queue://TEMPLATE_ORCHESTRATION_MESSAGES, transactionId = null, deliveryTime =
> 0, expiration = 0, timestamp = 1722565114517, arrival = 0, brokerInTime =
> 1722565319026, brokerOutTime = 1722565319026, correlationId = null, replyTo =
> null, persistent = true, type = null, priority = 4, groupID = null,
> groupSequence = 0, targetConsumerId = null, compressed = false, userID =
> null, content =
> org.apache.activemq.util.ByteSequence@222083bc<mailto:org.apache.activemq.util.ByteSequence@222083bc>,
> marshalledProperties =
> org.apache.activemq.util.ByteSequence@44247f43<mailto:org.apache.activemq.util.ByteSequence@44247f43>,
> dataStructure = null, redeliveryCounter = 3, size = 0, properties =
> {WORKSPACE_ID=66ac0e4ef78b094c390c1d6b,
> x__dash__datadog__dash__trace__dash__id=8195324751683450459,x__dash__datadog__dash__sampling__dash__priority=1,
>
> x__dash__datadog__dash__parent__dash__id=2836570378207542649,scheduledJobId=ID:ip-10-210-181-64-5463-1722503120609-7:1:50:1:529018,
>
> TEMPLATE_PURCHASE_ID=66ac0e4ef78b094c390c1d7b,tracestate=dd=t.dm:-1;t.tid:66ac41bd00000000,
> CALLER_REFERENCE_ID=66ac0e470cfa9827bc10744c,
> traceparent=00-66ac41bd0000000071bba470e30ef65b-275d85baed74bd79-01,
> redeliveryDelay=25000,x__dash__datadog__dash__tags=_dd.p.dm=-1,_dd.p.tid=66ac41bd00000000},
> readOnlyProperties = true, readOnlyBody = true, droppable = false,
> jmsXGroupFirstForConsumer = false}, redeliveryCounter = 3}
>
> We are using Amazon MQ with ActiveMQ version 5.16.3 (although we experienced
> this issue on previous versions too).
> Our java application is using org.apache.activemq:activemq-client:5.16.3.
>
> The Spring config is;
>
> @Bean
> public ConnectionFactory
> jmsConsumerConnectionFactory(@Value("${jms.broker.url}") String jmsBrokerUrl,
>
> @Value("${jms.broker.username}") String username,
>
> @Value("${jms.broker.password}") String password,
> RedeliveryPolicyMap
> consumerRedeliveryPolicyMap,
> ActiveMQPrefetchPolicy
> prefetchPolicy) {
> return activeMQConnectionFactory(jmsBrokerUrl, username, password,
> consumerRedeliveryPolicyMap, prefetchPolicy, "enc-consumer");
> }
>
> private ActiveMQConnectionFactory activeMQConnectionFactory(String
> jmsBrokerUrl, String username, String password, RedeliveryPolicyMap
> consumerRedeliveryPolicyMap, ActiveMQPrefetchPolicy prefetchPolicy, String
> clientIdPrefix) {
> ActiveMQConnectionFactory connectionFactory = new
> ActiveMQConnectionFactory(jmsBrokerUrl);
> connectionFactory.setUserName(trimToNull(username));
> connectionFactory.setPassword(trimToNull(password));
>
> connectionFactory.setObjectMessageSerializationDefered(true);
> connectionFactory.setAlwaysSyncSend(true);
> connectionFactory.setTrustAllPackages(true);
> connectionFactory.setRedeliveryPolicyMap(consumerRedeliveryPolicyMap);
> connectionFactory.setPrefetchPolicy(prefetchPolicy);
>
> connectionFactory.setClientIDPrefix(clientIdPrefix);
>
> return connectionFactory;
> }
>
>
> protected DefaultJmsListenerContainerFactory
> defaultJmsListenerContainerFactory(boolean autoStartupEnabled, String
> concurrency, ErrorHandler errorHandler) {
> DefaultJmsListenerContainerFactory factory = new
> DefaultJmsListenerContainerFactory();
> factory.setConcurrency(concurrency);
> factory.setConnectionFactory(jmsConsumerConnectionFactory);
> factory.setTransactionManager(localJmsTransactionManager);
> factory.setErrorHandler(errorHandler);
> factory.setAutoStartup(autoStartupEnabled);
> factory.setCacheLevel(CACHE_CONSUMER);
>
> return factory;
> }
>
> and the broker URI is typically;
>
>
> jms.broker.url="failover:(ssl://b-7cef9e05-66d6cbcf570b-1.mq.eu-west-2.amazonaws.com:61617,ssl://b-7cef9e05-66d6cbcf570b-2.mq.eu-west-2.amazonaws.com:61617)?initialReconnectDelay=100&maxReconnectAttempts=10&jms.checkForDuplicates=true&jms.watchTopicAdvisories=false&jms.prefetchPolicy.queuePrefetch=0&nested.wireFormat.maxInactivityDuration=1800000&nested.wireFormat.maxFrameSize=104857600"
>
> I’ve debugged the client code to try and see where it’s detecting the
> duplicate and it seems to have the duplicate message id in it’s cache. It’s
> difficult to reproduce and so it’s hard to tell if does indeed get the same
> message twice but given it doesn’t handle the message at all then I’m
> suspecting the cache is incorrectly preloaded. Could it be that it’s caching
> to the filesystem and when the application restarts it reloads the cache?
>
> Would appreciate any help as we really want duplicate detection enabled.
>
> Regards
> Ben
>
>
>
>
> [https://6032599.fs1.hubspotusercontent-na1.net/hubfs/6032599/Email%20images/Encompass%20logo/Encompass-(full)_300x52.png]
>
> The leader in Corporate Digital Identity
>
> Website<https://www.encompasscorporation.com/> |
> LinkedIn<https://www.linkedin.com/company/encompass-corporation/>
>
> Encompass Corporation UK Ltd | Company No. SC493055 | Address: Level 3, 33
> Bothwell Street, Glasgow, UK, G2 6NL
> Encompass Corporation Australia PTY Ltd | 605 414 499| Address: Level 1, 117
> Clarence Street, Sydney, New South Wales, 2000
> Encompass Corporation US Inc | Company No. 7946259 | Address: 11 West 42nd
> Street, New York, New York, 10036
>
> This email and any attachments is intended only for the use of the individual
> or entity named above and may contain confidential information. If you are
> not the intended recipient, any dissemination, distribution or copying of
> this email is prohibited. If received in error, please notify us immediately
> by return email and destroy the original message.
>
>
>
>
>
>
> Public
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscr...@activemq.apache.org
For additional commands, e-mail: users-h...@activemq.apache.org
For further information, visit: https://activemq.apache.org/contact
