Hello,
I am using Artemis 2.4.0 to broker messages through JMS queues/topics between a
set of clients. Some are Apache NMS 1.7.2 ActiveMQ clients and others are using
Artemis JMS client 1.5.4 included in Spring Boot 1.5.3. Broker topology is a
symmetric cluster of two live nodes with static connectors, both nodes having
been setup as replicating colocated backup pairs with scale down. I have two
quite frustrating issues at the moment: message redistribution not working
correctly and a memory leak causing eventual thread death.
ISSUE #1 - Message redistribution / load balancing not working:
Client 1 (NMS) connects to broker a and starts listening, artemis creates the
following address:
(Broker a):
A.B.*.C.D
|-queues
|-multicast
|-f64dd592-a8fb-442e-826d-927834d566f4
Server 1 (artemis-jms-client) connects to broker b and sends a message to
topic: A.B.f64dd592-a8fb-442e-826d-927834d566f4.C.D - this should be routed to
broker a since the corresponding queue has no consumers on broker b (the queue
does not exist). This however does not happen and the client receives no
messages. Broker b has some other clients connected, causing similar (but not
the same) queues having been created:
(Broker b):
A.B.*.C.D
|-queues
|-multicast
|-1eb48079-7fd8-40e9-b822-bcc25695ced0
|-9f295257-c352-4ae6-b74b-d5994f330485
ISSUE #2: - Memory leak and eventual thread death
Artemis broker has 4GB allocated heap space and global-max-size is set up as
half of that (being the default setting). Address-full-policy is set to PAGE
for all addresses and some individual addresses have small max-size-bytes
values set e.g. 104857600. As far as I know the paging settings should limit
memory usage but what happens is that at times Artemis uses the whole heap
space, encounters an out of memory error and dies:
05:39:29,510 WARN [org.eclipse.jetty.util.thread.QueuedThreadPool] :
java.lang.OutOfMemoryError: Java heap space
05:39:16,646 WARN [io.netty.channel.ChannelInitializer] Failed to initialize a
channel. Closing: [id: ...]: java.lang.OutOfMemoryError: Java heap space
05:41:05,597 WARN [org.eclipse.jetty.util.thread.QueuedThreadPool] Unexpected
thread death: org.eclipse.jetty.util.thread.QueuedThreadPool$2@5ffaba31 in
qtp20111564{STARTED,8<=8<=200,i=2,q=0}
Are these known issues in Artemis or misconfigurations in the brokers?
The broker configurations are as follows. Broker b has an identical
configuration excluding that the cluster connector's connector-ref and
static-connector connector-ref refer to broker b and broker a respectively.
Best regards,
broker.xml (broker a):
<?xml version='1.0'?>
<configuration xmlns="urn:activemq"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance";
xsi:schemaLocation="urn:activemq /schema/artemis-configuration.xsd">
<core xmlns="urn:activemq:core"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance";
xsi:schemaLocation="urn:activemq:core ">
<name>[broker-a-ip]</name>
<persistence-enabled>true</persistence-enabled>
<journal-type>NIO</journal-type>
<paging-directory>...</paging-directory>
<bindings-directory>...</bindings-directory>
<journal-directory>...</journal-directory>
<large-messages-directory>...</large-messages-directory>
<journal-datasync>true</journal-datasync>
<journal-min-files>2</journal-min-files>
<journal-pool-files>-1</journal-pool-files>
<journal-buffer-timeout>788000</journal-buffer-timeout>
<disk-scan-period>5000</disk-scan-period>
<max-disk-usage>97</max-disk-usage>
<critical-analyzer>true</critical-analyzer>
<critical-analyzer-timeout>120000</critical-analyzer-timeout>
<critical-analyzer-check-period>60000</critical-analyzer-check-period>
<critical-analyzer-policy>HALT</critical-analyzer-policy>
<acceptors>
<acceptor name="invm-acceptor">vm://0</acceptor>
<acceptor name="artemis">tcp://0.0.0.0:61616</acceptor>
<acceptor
name="ssl">tcp://0.0.0.0:61617?sslEnabled=true;keyStorePath=...;keyStorePassword=...</acceptor>
</acceptors>
<connectors>
<connector name="invm-connector">vm://0</connector>
<connector
name="netty-connector">tcp://[broker-a-ip]:61616</connector>
<connector name="broker-b-connector">[broker-b-ip]:61616</connector>
</connectors>
<cluster-connections>
<cluster-connection name="cluster-name">
<address>*</address>
<connector-ref>netty-connector</connector-ref>
<retry-interval>500</retry-interval>
<reconnect-attempts>5</reconnect-attempts>
<use-duplicate-detection>true</use-duplicate-detection>
<message-load-balancing>ON_DEMAND</message-load-balancing>
<max-hops>1</max-hops>
<static-connectors>
<connector-ref>broker-b-connector</connector-ref>
</static-connectors>
</cluster-connection>
</cluster-connections>
<ha-policy>
<replication>
<colocated>
<backup-request-retry-interval>5000</backup-request-retry-interval>
<max-backups>3</max-backups>
<request-backup>true</request-backup>
<backup-port-offset>100</backup-port-offset>
<excludes>
<connector-ref>invm-connector</connector-ref>
<connector-ref>netty-connector</connector-ref>
</excludes>
<master>
<check-for-live-server>true</check-for-live-server>
</master>
<slave>
<restart-backup>false</restart-backup>
<scale-down />
</slave>
</colocated>
</replication>
</ha-policy>
<cluster-user>ARTEMIS.CLUSTER.ADMIN.USER</cluster-user>
<cluster-password>[the shared cluster password]</cluster-password>
<security-settings>
<security-setting match="#">
<permission type="createDurableQueue" roles="amq, other-role" />
<permission type="deleteDurableQueue" roles="amq, other-role" />
<permission type="createNonDurableQueue" roles="amq,
other-role" />
<permission type="createAddress" roles="amq, other-role" />
<permission type="deleteNonDurableQueue" roles="amq,
other-role" />
<permission type="deleteAddress" roles="amq, other-role" />
<permission type="consume" roles="amq, other-role" />
<permission type="browse" roles="amq, other-role" />
<permission type="send" roles="amq, other-role" />
<permission type="manage" roles="amq" />
</security-setting>
<security-setting match="A.some.queue">
<permission type="createNonDurableQueue" roles="amq,
other-role" />
<permission type="deleteNonDurableQueue" roles="amq,
other-role" />
<permission type="createDurableQueue" roles="amq, other-role" />
<permission type="deleteDurableQueue" roles="amq, other-role" />
<permission type="createAddress" roles="amq, other-role" />
<permission type="deleteAddress" roles="amq, other-role" />
<permission type="consume" roles="amq, other-role" />
<permission type="browse" roles="amq, other-role" />
<permission type="send" roles="amq, other-role" />
</security-setting>
<security-setting match="A.some.other.queue">
<permission type="createNonDurableQueue" roles="amq,
other-role" />
<permission type="deleteNonDurableQueue" roles="amq,
other-role" />
<permission type="createDurableQueue" roles="amq, other-role" />
<permission type="deleteDurableQueue" roles="amq, other-role" />
<permission type="createAddress" roles="amq, other-role" />
<permission type="deleteAddress" roles="amq, other-role" />
<permission type="consume" roles="amq, other-role" />
<permission type="browse" roles="amq, other-role" />
<permission type="send" roles="amq, other-role" />
</security-setting>
...
... etc.
...
</security-settings>
<address-settings>
<address-setting match="activemq.management#">
<dead-letter-address>DLQ</dead-letter-address>
<expiry-address>ExpiryQueue</expiry-address>
<redelivery-delay>0</redelivery-delay>
<max-size-bytes>-1</max-size-bytes>
<message-counter-history-day-limit>10</message-counter-history-day-limit>
<address-full-policy>PAGE</address-full-policy>
</address-setting>
<!--default for catch all -->
<address-setting match="#">
<dead-letter-address>DLQ</dead-letter-address>
<expiry-address>ExpiryQueue</expiry-address>
<redelivery-delay>0</redelivery-delay>
<max-size-bytes>-1</max-size-bytes>
<message-counter-history-day-limit>10</message-counter-history-day-limit>
<address-full-policy>PAGE</address-full-policy>
<redistribution-delay>1000</redistribution-delay>
</address-setting>
<address-setting match="DLQ">
<!-- 100 * 1024 * 1024 -> 100MB -->
<max-size-bytes>104857600</max-size-bytes>
<!-- 1000 * 60 * 60 -> 1h -->
<expiry-delay>3600000</expiry-delay>
<expiry-address />
</address-setting>
<address-setting match="A.some.queue">
<redelivery-delay-multiplier>1.0</redelivery-delay-multiplier>
<redelivery-delay>0</redelivery-delay>
<max-redelivery-delay>10</max-redelivery-delay>
</address-setting>
<address-setting match="A.some.other.queue">
<redelivery-delay-multiplier>1.0</redelivery-delay-multiplier>
<redelivery-delay>0</redelivery-delay>
<max-redelivery-delay>10</max-redelivery-delay>
<max-delivery-attempts>1</max-delivery-attempts>
<max-size-bytes>104857600</max-size-bytes>
</address-setting>
...
... etc.
...
</address-settings>
<addresses>
<address name="DLQ">
<anycast>
<queue name="DLQ" />
</anycast>
</address>
<address name="ExpiryQueue">
<anycast>
<queue name="ExpiryQueue" />
</anycast>
</address>
<address name="A.some.queue">
<anycast>
<queue name="A.some.queue">
<durable>true</durable>
</queue>
</anycast>
</address>
<address name="A.some.other.queue">
<anycast>
<queue name="A.some.other.queue">
<durable>true</durable>
</queue>
</anycast>
</address>
...
... etc.
...
</addresses>
</core>
</configuration>
