Hi, we are using Artemis with the following setup: - 2 independent broker instances (on 2 hosts)- a cluster configuration to create a Core bridge between both instances (no failover, no HA)
- multiple JMS clients produce and consume AMQP messages using topics - the clients do a failover themself - Artemis versions (2.21.0, 2.29.0-SNAPSHOT cloned on 08.03)
Every thing is working fine. Independent of the Artemis instance the producer or consumers are connected to they receive all messages in the order of creation.
To simulate a server failure we kill (-9) Artemis instance 1 and restart the instance again (~ 1/2 minute later).
- 1 producer connects to the restarted instance 1 - multiple consumers are (still) connected to instance 2 - 1 consumer connects to the restarted instance 1 The producer sends messages with a delay of 1 ms. Now we see that- the order of messages received by the consumer connected to instance 1 frequently does not match the order the messages are created - the order of messages received by consumers connected to instance 2 matches the order the messages are created
It is essential for us that the messages arrive in the order of creation. Do you have any ideas what went wrong or we are doing wrong? Thanks in advance, OliverPls note: the attached files are used to reproduce what we saw in production.
This test configuration uses 1 docker instance per Artemis broker. Both instances are running on the same host using different ports.
<?xml version='1.0'?> <!-- Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements. See the NOTICE file distributed with this work for additional information regarding copyright ownership. The ASF licenses this file to you under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. --> <configuration xmlns="urn:activemq" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; xmlns:xi="http://www.w3.org/2001/XInclude"; xsi:schemaLocation="urn:activemq /schema/artemis-configuration.xsd"> <core xmlns="urn:activemq:core" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; xsi:schemaLocation="urn:activemq:core "> <name>0.0.0.0</name> <persistence-enabled>false</persistence-enabled> <!-- this could be ASYNCIO, MAPPED, NIO ASYNCIO: Linux Libaio MAPPED: mmap files NIO: Plain Java Files --> <journal-type>ASYNCIO</journal-type> <paging-directory>data/paging</paging-directory> <bindings-directory>data/bindings</bindings-directory> <journal-directory>data/journal</journal-directory> <large-messages-directory>data/large-messages</large-messages-directory> <journal-datasync>true</journal-datasync> <journal-min-files>2</journal-min-files> <journal-pool-files>10</journal-pool-files> <journal-device-block-size>4096</journal-device-block-size> <journal-file-size>10M</journal-file-size> <!-- This value was determined through a calculation. Your system could perform 7.58 writes per millisecond on the current journal configuration. That translates as a sync write every 132000 nanoseconds. Note: If you specify 0 the system will perform writes directly to the disk. We recommend this to be 0 if you are using journalType=MAPPED and journal-datasync=false. --> <journal-buffer-timeout>132000</journal-buffer-timeout> <!-- When using ASYNCIO, this will determine the writing queue depth for libaio. --> <journal-max-io>4096</journal-max-io> <!-- how often we are looking for how many bytes are being used on the disk in ms --> <disk-scan-period>5000</disk-scan-period> <!-- once the disk hits this limit the system will block, or close the connection in certain protocols that won't support flow control. --> <max-disk-usage>99</max-disk-usage> <!-- should the broker detect dead locks and other issues --> <critical-analyzer>true</critical-analyzer> <critical-analyzer-timeout>120000</critical-analyzer-timeout> <critical-analyzer-check-period>60000</critical-analyzer-check-period> <critical-analyzer-policy>HALT</critical-analyzer-policy> <page-sync-timeout>184000</page-sync-timeout> <!-- the system will enter into page mode once you hit this limit. This is an estimate in bytes of how much the messages are using in memory The system will use half of the available memory (-Xmx) by default for the global-max-size. You may specify a different value here if you need to customize it to your needs. <global-max-size>100Mb</global-max-size> --> <!-- the maximum number of messages accepted before entering full address mode. if global-max-size is specified the full address mode will be specified by whatever hits it first. --> <global-max-messages>-1</global-max-messages> <acceptors> <!-- Acceptor for every supported protocol --> <acceptor name="artemis">tcp://0.0.0.0:61611?tcpSendBufferSize=1048576;tcpReceiveBufferSize=1048576;amqpMinLargeMessageSize=102400;protocols=CORE;useEpoll=true;amqpCredits=1000;amqpLowCredits=300;amqpDuplicateDetection=true;supportAdvisory=false;suppressInternalManagementObjects=false</acceptor> <!-- AMQP Acceptor. Listens on default AMQP port for AMQP traffic.--> <acceptor name="amqp_internal">tcp://0.0.0.0:5671?tcpSendBufferSize=1048576;tcpReceiveBufferSize=1048576;protocols=AMQP;useEpoll=true;amqpCredits=1000;amqpLowCredits=300;amqpMinLargeMessageSize=102400;amqpDuplicateDetection=true</acceptor> </acceptors> <connectors xmlns="urn:activemq:core"> <connector name="br-own-connector">tcp://172.18.104.235:61611</connector> <connector name="br-2-connector">tcp://172.18.104.235:61612</connector> </connectors> <cluster-connections xmlns="urn:activemq:core"> <cluster-connection name="broker-cluster"> <connector-ref>br-own-connector</connector-ref> <retry-interval>500</retry-interval> <use-duplicate-detection>true</use-duplicate-detection> <message-load-balancing>ON_DEMAND</message-load-balancing> <max-hops>1</max-hops> <static-connectors> <connector-ref>br-2-connector</connector-ref> </static-connectors> </cluster-connection> </cluster-connections> <security-settings> <security-setting match="#"> <permission type="createNonDurableQueue" roles="amq"/> <permission type="deleteNonDurableQueue" roles="amq"/> <permission type="createDurableQueue" roles="amq"/> <permission type="deleteDurableQueue" roles="amq"/> <permission type="createAddress" roles="amq"/> <permission type="deleteAddress" roles="amq"/> <permission type="consume" roles="amq"/> <permission type="browse" roles="amq"/> <permission type="send" roles="amq"/> <!-- we need this otherwise ./artemis data imp wouldn't work --> <permission type="manage" roles="amq"/> </security-setting> </security-settings> <address-settings> <!-- if you define auto-create on certain queues, management has to be auto-create --> <address-setting match="activemq.management#"> <dead-letter-address>DLQ</dead-letter-address> <expiry-address>ExpiryQueue</expiry-address> <redelivery-delay>0</redelivery-delay> <!-- with -1 only the global-max-size is in use for limiting --> <max-size-bytes>-1</max-size-bytes> <message-counter-history-day-limit>10</message-counter-history-day-limit> <address-full-policy>PAGE</address-full-policy> <auto-create-queues>true</auto-create-queues> <auto-create-addresses>true</auto-create-addresses> <auto-delete-queues>true</auto-delete-queues> <auto-delete-addresses>true</auto-delete-addresses> </address-setting> <!--default for catch all--> <address-setting match="#"> <dead-letter-address>DLQ</dead-letter-address> <expiry-address>ExpiryQueue</expiry-address> <redelivery-delay>0</redelivery-delay> <!-- if max-size-bytes and max-size-messages were both enabled, the system will enter into paging based on the first attribute to hits the maximum value --> <!-- limit for the address in bytes, -1 means unlimited --> <max-size-bytes>-1</max-size-bytes> <!-- limit for the address in messages, -1 means unlimited --> <max-size-messages>-1</max-size-messages> <!-- the size of each file on paging. Notice we keep files in memory while they are in use. Lower this setting if you have too many queues in memory. --> <page-size-bytes>10M</page-size-bytes> <!-- limit how many messages are read from paging into the Queue. --> <max-read-page-messages>-1</max-read-page-messages> <!-- limit how much memory is read from paging into the Queue. --> <max-read-page-bytes>20M</max-read-page-bytes> <message-counter-history-day-limit>10</message-counter-history-day-limit> <address-full-policy>PAGE</address-full-policy> <auto-create-queues>true</auto-create-queues> <auto-create-addresses>true</auto-create-addresses> <auto-delete-queues>false</auto-delete-queues> <auto-delete-addresses>false</auto-delete-addresses> </address-setting> </address-settings> <addresses> <address name="DLQ"> <anycast> <queue name="DLQ" /> </anycast> </address> <address name="ExpiryQueue"> <anycast> <queue name="ExpiryQueue" /> </anycast> </address> </addresses> <!-- Uncomment the following if you want to use the Standard LoggingActiveMQServerPlugin pluging to log in events <broker-plugins> <broker-plugin class-name="org.apache.activemq.artemis.core.server.plugin.impl.LoggingActiveMQServerPlugin"> <property key="LOG_ALL_EVENTS" value="true"/> <property key="LOG_CONNECTION_EVENTS" value="true"/> <property key="LOG_SESSION_EVENTS" value="true"/> <property key="LOG_CONSUMER_EVENTS" value="true"/> <property key="LOG_DELIVERING_EVENTS" value="true"/> <property key="LOG_SENDING_EVENTS" value="true"/> <property key="LOG_INTERNAL_EVENTS" value="true"/> </broker-plugin> </broker-plugins> --> </core> </configuration>
<?xml version='1.0'?> <!-- Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements. See the NOTICE file distributed with this work for additional information regarding copyright ownership. The ASF licenses this file to you under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. --> <configuration xmlns="urn:activemq" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; xmlns:xi="http://www.w3.org/2001/XInclude"; xsi:schemaLocation="urn:activemq /schema/artemis-configuration.xsd"> <core xmlns="urn:activemq:core" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; xsi:schemaLocation="urn:activemq:core "> <name>0.0.0.0</name> <persistence-enabled>false</persistence-enabled> <!-- this could be ASYNCIO, MAPPED, NIO ASYNCIO: Linux Libaio MAPPED: mmap files NIO: Plain Java Files --> <journal-type>ASYNCIO</journal-type> <paging-directory>data/paging</paging-directory> <bindings-directory>data/bindings</bindings-directory> <journal-directory>data/journal</journal-directory> <large-messages-directory>data/large-messages</large-messages-directory> <journal-datasync>true</journal-datasync> <journal-min-files>2</journal-min-files> <journal-pool-files>10</journal-pool-files> <journal-device-block-size>4096</journal-device-block-size> <journal-file-size>10M</journal-file-size> <!-- This value was determined through a calculation. Your system could perform 8.33 writes per millisecond on the current journal configuration. That translates as a sync write every 119999 nanoseconds. Note: If you specify 0 the system will perform writes directly to the disk. We recommend this to be 0 if you are using journalType=MAPPED and journal-datasync=false. --> <journal-buffer-timeout>132000</journal-buffer-timeout> <!-- When using ASYNCIO, this will determine the writing queue depth for libaio. --> <journal-max-io>4096</journal-max-io> <!-- how often we are looking for how many bytes are being used on the disk in ms --> <disk-scan-period>5000</disk-scan-period> <!-- once the disk hits this limit the system will block, or close the connection in certain protocols that won't support flow control. --> <max-disk-usage>99</max-disk-usage> <!-- should the broker detect dead locks and other issues --> <critical-analyzer>true</critical-analyzer> <critical-analyzer-timeout>120000</critical-analyzer-timeout> <critical-analyzer-check-period>60000</critical-analyzer-check-period> <critical-analyzer-policy>HALT</critical-analyzer-policy> <page-sync-timeout>184000</page-sync-timeout> <!-- the system will enter into page mode once you hit this limit. This is an estimate in bytes of how much the messages are using in memory The system will use half of the available memory (-Xmx) by default for the global-max-size. You may specify a different value here if you need to customize it to your needs. <global-max-size>100Mb</global-max-size> --> <!-- the maximum number of messages accepted before entering full address mode. if global-max-size is specified the full address mode will be specified by whatever hits it first. --> <global-max-messages>-1</global-max-messages> <acceptors> <!-- Acceptor for every supported protocol --> <acceptor name="artemis">tcp://0.0.0.0:61612?tcpSendBufferSize=1048576;tcpReceiveBufferSize=1048576;amqpMinLargeMessageSize=102400;protocols=CORE;useEpoll=true;amqpCredits=1000;amqpLowCredits=300;amqpDuplicateDetection=true;supportAdvisory=false;suppressInternalManagementObjects=false</acceptor> <!-- AMQP Acceptor. Listens on default AMQP port for AMQP traffic.--> <acceptor name="amqp_internal">tcp://0.0.0.0:5672?tcpSendBufferSize=1048576;tcpReceiveBufferSize=1048576;protocols=AMQP;useEpoll=true;amqpCredits=1000;amqpLowCredits=300;amqpMinLargeMessageSize=102400;amqpDuplicateDetection=true</acceptor> </acceptors> <connectors xmlns="urn:activemq:core"> <connector name="br-own-connector">tcp://172.18.104.235:61612</connector> <connector name="br-1-connector">tcp://172.18.104.235:61611</connector> </connectors> <cluster-connections xmlns="urn:activemq:core"> <cluster-connection name="broker-cluster"> <connector-ref>br-own-connector</connector-ref> <retry-interval>500</retry-interval> <use-duplicate-detection>true</use-duplicate-detection> <message-load-balancing>ON_DEMAND</message-load-balancing> <max-hops>1</max-hops> <static-connectors> <connector-ref>br-1-connector</connector-ref> </static-connectors> </cluster-connection> </cluster-connections> <security-settings> <security-setting match="#"> <permission type="createNonDurableQueue" roles="amq"/> <permission type="deleteNonDurableQueue" roles="amq"/> <permission type="createDurableQueue" roles="amq"/> <permission type="deleteDurableQueue" roles="amq"/> <permission type="createAddress" roles="amq"/> <permission type="deleteAddress" roles="amq"/> <permission type="consume" roles="amq"/> <permission type="browse" roles="amq"/> <permission type="send" roles="amq"/> <!-- we need this otherwise ./artemis data imp wouldn't work --> <permission type="manage" roles="amq"/> </security-setting> </security-settings> <address-settings> <!-- if you define auto-create on certain queues, management has to be auto-create --> <address-setting match="activemq.management#"> <dead-letter-address>DLQ</dead-letter-address> <expiry-address>ExpiryQueue</expiry-address> <redelivery-delay>0</redelivery-delay> <!-- with -1 only the global-max-size is in use for limiting --> <max-size-bytes>-1</max-size-bytes> <message-counter-history-day-limit>10</message-counter-history-day-limit> <address-full-policy>PAGE</address-full-policy> <auto-create-queues>true</auto-create-queues> <auto-create-addresses>true</auto-create-addresses> <auto-delete-queues>true</auto-delete-queues> <auto-delete-addresses>true</auto-delete-addresses> </address-setting> <!--default for catch all--> <address-setting match="#"> <dead-letter-address>DLQ</dead-letter-address> <expiry-address>ExpiryQueue</expiry-address> <redelivery-delay>0</redelivery-delay> <!-- if max-size-bytes and max-size-messages were both enabled, the system will enter into paging based on the first attribute to hits the maximum value --> <!-- limit for the address in bytes, -1 means unlimited --> <max-size-bytes>-1</max-size-bytes> <!-- limit for the address in messages, -1 means unlimited --> <max-size-messages>-1</max-size-messages> <!-- the size of each file on paging. Notice we keep files in memory while they are in use. Lower this setting if you have too many queues in memory. --> <page-size-bytes>10M</page-size-bytes> <!-- limit how many messages are read from paging into the Queue. --> <max-read-page-messages>-1</max-read-page-messages> <!-- limit how much memory is read from paging into the Queue. --> <max-read-page-bytes>20M</max-read-page-bytes> <message-counter-history-day-limit>10</message-counter-history-day-limit> <address-full-policy>PAGE</address-full-policy> <auto-create-queues>true</auto-create-queues> <auto-create-addresses>true</auto-create-addresses> <auto-delete-queues>false</auto-delete-queues> <auto-delete-addresses>false</auto-delete-addresses> </address-setting> </address-settings> <addresses> <address name="DLQ"> <anycast> <queue name="DLQ" /> </anycast> </address> <address name="ExpiryQueue"> <anycast> <queue name="ExpiryQueue" /> </anycast> </address> </addresses> <!-- Uncomment the following if you want to use the Standard LoggingActiveMQServerPlugin pluging to log in events <broker-plugins> <broker-plugin class-name="org.apache.activemq.artemis.core.server.plugin.impl.LoggingActiveMQServerPlugin"> <property key="LOG_ALL_EVENTS" value="true"/> <property key="LOG_CONNECTION_EVENTS" value="true"/> <property key="LOG_SESSION_EVENTS" value="true"/> <property key="LOG_CONSUMER_EVENTS" value="true"/> <property key="LOG_DELIVERING_EVENTS" value="true"/> <property key="LOG_SENDING_EVENTS" value="true"/> <property key="LOG_INTERNAL_EVENTS" value="true"/> </broker-plugin> </broker-plugins> --> </core> </configuration>
artemis-broker-logs.tar.gz
Description: application/gzip
