Hello,
I'm using Apache Artemis as MQTT broker for our IOT projects.
It's a clean Artemis installation of version 1.5.1., on a server (CentOS 7)
which has 2 vCPU, 8 GB of RAM (4GB of Heap Space dedicated to Artemis) and 50
GB of SSD data disk.
After the installation of artemis Broker we started to test it with about 10-15
clients constantly connected, 150 subscribed topics and an average of 2
messages per minute per client. I think these are not huge numbers, right?
For the few days after the installation, all was good and the broker worked
perfectly: it was fast and reliable. But day by day performances have decreased
and after about 10 days of usage it is became almost unusable: due to resources
consumption. The following is my "top" situation on the server:
top - 08:59:50 up 2 days, 22:09, 1 user, load average: 2.56, 2.88, 3.00
Tasks: 92 total, 1 running, 91 sleeping, 0 stopped, 0 zombie
%Cpu(s): 99.5 us, 0.5 sy, 0.0 ni, 0.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
KiB Mem : 7747288 total, 290968 free, 4697976 used, 2758344 buff/cache
KiB Swap: 0 total, 0 free, 0 used. 2734768 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
4352 mqtt 20 0 7043380 4.348g 15192 S 200.0 58.9 6153:54 java
1 root 20 0 128096 5176 2420 S 0.0 0.1 0:04.01 systemd
Now I can see the broker starting to page on disk almost always. For sure it's
a wrong configuration of ours. Currently, it seems that very old address/queue
(and the related retained messages) are always kept, making memory and cpu
consuption growing more and more. And even after a restart, the broker takes so
much to get up. Before to become reachable, it starts to make many management
operations like, for example, retrieve data from paging, etc.. But I also see
the broker that starts to register old topics and queues we don't need. How to
clean them? How to make a topic/queue expire?
Inside my broker.xml I did set up the parameter last-value-queue=true, thinking
that this was the problem ....but it doesn't solve. Or better: probably it's me
that I've not understood the correct meaning of the parameter. I did check also
the clients' parameters when they connect to the broker, to be sure they don't
set, for example, clean-session = false (saying the broker to keep all messages
also when a client disconnects). But they make it in the right way. The only
thing is that they don't specify a client-id. They connect by using a
username/password and certificate (over tls). So, every time a client connects,
Artemis automatically provide a random client-id for it (if I understood well).
Attached you can find my broker.xml configuration file: it's pretty much the
same default created during the installation procedure, but for the acceptors
(which I've customized for my MQTT purpose) and the addition of parameter
last-value-queue = true inside the address-setting section.
Please: some of you could help me? How I have to configure my broker instance
to understand and solve these performance issues?
Thanks in advance
Francesco
________________________________
Ce message, ainsi que tous les fichiers joints à ce message, peuvent contenir
des informations sensibles et/ ou confidentielles ne devant pas être
divulguées. Si vous n'êtes pas le destinataire de ce message (ou que vous
recevez ce message par erreur), nous vous remercions de le notifier
immédiatement à son expéditeur, et de détruire ce message. Toute copie,
divulgation, modification, utilisation ou diffusion, non autorisée, directe ou
indirecte, de tout ou partie de ce message, est strictement interdite.
This e-mail, and any document attached hereby, may contain confidential and/or
privileged information. If you are not the intended recipient (or have received
this e-mail in error) please notify the sender immediately and destroy this
e-mail. Any unauthorized, direct or indirect, copying, disclosure, distribution
or other use of the material or parts thereof is strictly forbidden.
<?xml version='1.0'?>
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-->
<configuration xmlns="urn:activemq"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance";
xsi:schemaLocation="urn:activemq /schema/artemis-configuration.xsd">
<jms xmlns="urn:activemq:jms">
<queue name="DLQ"/>
<queue name="ExpiryQueue"/>
</jms>
<core xmlns="urn:activemq:core">
<name>0.0.0.0</name>
<!-- <persistence-enabled>true</persistence-enabled> -->
<persistence-enabled>true</persistence-enabled>
<!-- this could be ASYNCIO or NIO
-->
<journal-type>ASYNCIO</journal-type>
<paging-directory>./data/paging</paging-directory>
<bindings-directory>./data/bindings</bindings-directory>
<journal-directory>./data/journal</journal-directory>
<large-messages-directory>./data/large-messages</large-messages-directory>
<journal-datasync>true</journal-datasync>
<journal-min-files>2</journal-min-files>
<journal-pool-files>-1</journal-pool-files>
<!--
You can specify the NIC you want to use to verify if the network
<network-check-NIC>theNickName</network-check-NIC>
-->
<!--
Use this to use an HTTP server to validate the network
<network-check-URL-list>http://www.apache.org</network-check-URL-list> -->
<!-- <network-check-period>10000</network-check-period> -->
<!-- <network-check-timeout>1000</network-check-timeout> -->
<!-- this is a comma separated list, no spaces, just DNS or IPs
it should accept IPV6
Warning: Make sure you understand your network topology as this is meant to validate if your network is valid.
Using IPs that could eventually disappear or be partially visible may defeat the purpose.
You can use a list of multiple IPs, and if any successful ping will make the server OK to continue running -->
<!-- <network-check-list>10.0.0.1</network-check-list> -->
<!-- use this to customize the ping used for ipv4 addresses -->
<!-- <network-check-ping-command>ping -c 1 -t %d %s</network-check-ping-command> -->
<!-- use this to customize the ping used for ipv6 addresses -->
<!-- <network-check-ping6-command>ping6 -c 1 %2$s</network-check-ping6-command> -->
<!--
This value was determined through a calculation.
Your system could perform 1.91 writes per millisecond
on the current journal configuration.
That translates as a sync write every 524000 nanoseconds
-->
<journal-buffer-timeout>6468000</journal-buffer-timeout>
<!-- how often we are looking for how many bytes are being used on the disk in ms -->
<disk-scan-period>5000</disk-scan-period>
<!-- once the disk hits this limit the system will block, or close the connection in certain protocols
that won't support flow control. -->
<max-disk-usage>90</max-disk-usage>
<!-- the system will enter into page mode once you hit this limit.
This is an estimate in bytes of how much the messages are using in memory -->
<global-max-size>104857600</global-max-size>
<acceptors>
<!-- Acceptor for every supported protocol -->
<acceptor name="artemis">tcp://0.0.0.0:61616?tcpSendBufferSize=1048576;tcpReceiveBufferSize=1048576;protocols=CORE,AMQP,STOMP,HORNETQ,MQTT,OPENWIRE</acceptor>
<!-- AMQP Acceptor. Listens on default AMQP port for AMQP traffic.-->
<!-- <acceptor name="amqp">tcp://0.0.0.0:5672?protocols=AMQP</acceptor> -->
<!-- STOMP Acceptor. -->
<!-- <acceptor name="stomp">tcp://0.0.0.0:61613?protocols=STOMP</acceptor> -->
<!-- HornetQ Compatibility Acceptor. Enables HornetQ Core and STOMP for legacy HornetQ clients. -->
<!-- <acceptor name="hornetq">tcp://0.0.0.0:5445?protocols=HORNETQ,STOMP</acceptor> -->
<!-- MQTT Acceptor -->
<!-- <acceptor name="mqtt">tcp://0.0.0.0:1883?protocols=MQTT</acceptor> -->
<acceptor name="mqtt">tcp://172.16.2.5:1883?protocols=MQTT</acceptor>
<acceptor name="mqtts">tcp://172.16.2.5:8883?protocols=MQTT;sslEnabled=true;needClientAuth=true;keyStorePath=/app/mqtt/activeMQ/brokers/eltbroker01/tls/brokerkeystore;keyStorePassword=changeit;trustStorePath=/app/mqtt/activeMQ/brokers/eltbroker01/tls/brokertruststore;trustStorePassword=changeit</acceptor>
<acceptor name="mqtts2">tcp://172.16.2.5:8884?protocols=MQTT;sslEnabled=true;needClientAuth=false;keyStorePath=/app/mqtt/activeMQ/brokers/eltbroker01/tls/brokerkeystore;keyStorePassword=changeit;trustStorePath=/app/mqtt/activeMQ/brokers/eltbroker01/tls/brokertruststore;trustStorePassword=changeit</acceptor>
</acceptors>
<security-settings>
<security-setting match="#">
<permission type="createNonDurableQueue" roles="mqtt"/>
<permission type="deleteNonDurableQueue" roles="mqtt"/>
<permission type="createDurableQueue" roles="mqtt"/>
<permission type="deleteDurableQueue" roles="mqtt"/>
<permission type="consume" roles="mqtt"/>
<permission type="browse" roles="mqtt"/>
<permission type="send" roles="mqtt"/>
<!-- we need this otherwise ./artemis data imp wouldn't work -->
<permission type="manage" roles="mqtt"/>
</security-setting>
</security-settings>
<address-settings>
<!--default for catch all-->
<address-setting match="#">
<dead-letter-address>jms.queue.DLQ</dead-letter-address>
<expiry-address>jms.queue.ExpiryQueue</expiry-address>
<!-- <expiry-delay>3000</expiry-delay> -->
<redelivery-delay>0</redelivery-delay>
<!-- with -1 only the global-max-size is in use for limiting -->
<max-size-bytes>-1</max-size-bytes>
<message-counter-history-day-limit>10</message-counter-history-day-limit>
<address-full-policy>PAGE</address-full-policy>
<last-value-queue>true</last-value-queue>
</address-setting>
</address-settings>
</core>
</configuration>
