I have a new instance of Artemis now two weeks in a production scenario.
I'm using it as part of an integration layer between a website, inventory
manager, and an ERP. The website published to an address where I have an
interface pulling the messages from a queue and processing them through an
API to the inventory manager. Order updates are pulled from the inventory
manager and placed on an Artemis address. An interface pulls from a queue
and sends them back to the website via another API.
I'm using a shared file store HA pair. The shared store is nfs.
It seems that after I restart Artemis, it continues to consume heap space.
It goes up and down but trends up. This morning it got rather high, then
made several big jumps, spikes and valleys between 4G and 6G, and then even
2G and 5G. The CPU utilization became very high too. I ended up restarting
Artemis before the website got too busy.
Looking at the logs I see a number of java errors.
## I get a number of these bellow:
IO Error completing the transaction, code = 6, message = Cannot find tx
with id 17275717863
2021-02-06 00:19:56,294 ERROR
[org.apache.activemq.artemis.core.journal.impl.JournalImpl]
appendRollbackRecord:java.lang.IllegalStateException: Cannot find tx with
id 17275720650: java.lang.IllegalStateException: Cannot find tx with id
17275720650
at
org.apache.activemq.artemis.core.journal.impl.JournalImpl$10.run(JournalImpl.java:1451)
[artemis-journal-2.14.0.jar:2.14.0]
at
org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:42)
[artemis-commons-2.14.0.jar:2.14.0]
at
org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:31)
[artemis-commons-2.14.0.jar:2.14.0]
at
org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:65)
[artemis-commons-2.14.0.jar:2.14.0]
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
[java.base:]
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
[java.base:]
at
org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
[artemis-commons-2.14.0.jar:2.14.0]
## Then I get a whole bunch of these:
Connection failure to /{client IPs}:46691 has been detected: AMQ229014: Did
not receive data from /52.179.166.52:46691 within the 30,000ms connection
TTL. The connection will now be closed. [code=CONNECTION_TIMEDOUT]
## Then I get a bunch of these, some hundres:
2021-02-06 05:57:09,839 WARN [org.apache.activemq.artemis.core.server]
AMQ222104: IO Error completing the transaction, code = 6, message = Cannot
find tx with id 17278164299
2021-02-06 05:57:09,839 WARN [org.apache.activemq.artemis.core.server]
AMQ222104: IO Error completing the transaction, code = 6, message = Cannot
find tx with id 17278164299
2021-02-06 05:57:09,839 WARN [org.apache.activemq.artemis.core.server]
AMQ222104: IO Error completing the transaction, code = 6, message = Cannot
find tx with id 17278164299
2021-02-06 05:57:09,839 WARN [org.apache.activemq.artemis.core.server]
AMQ222104: IO Error completing the transaction, code = 6, message = Cannot
find tx with id 17278164299
2021-02-06 05:57:09,839 WARN [org.apache.activemq.artemis.core.server]
AMQ222104: IO Error completing the transaction, code = 6, message = Cannot
find tx with id 17278164299
2021-02-06 05:57:09,839 WARN [org.apache.activemq.artemis.core.server]
AMQ222104: IO Error completing the transaction, code = 6, message = Cannot
find tx with id 17278164299
2021-02-06 05:57:09,839 WARN [org.apache.activemq.artemis.core.server]
AMQ222104: IO Error completing the transaction, code = 6, message = Cannot
find tx with id 17278164299
2021-02-06 06:11:39,858 ERROR
[org.apache.activemq.artemis.core.journal.impl.JournalImpl]
appendRollbackRecord:java.lang.IllegalStateException: Cannot find tx with
id 17278212087: java.lang.IllegalStateException: Cannot find tx with id
17278212087
at
org.apache.activemq.artemis.core.journal.impl.JournalImpl$10.run(JournalImpl.java:1451)
[artemis-journal-2.14.0.jar:2.14.0]
at
org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:42)
[artemis-commons-2.14.0.jar:2.14.0]
at
org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:31)
[artemis-commons-2.14.0.jar:2.14.0]
at
org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:65)
[artemis-commons-2.14.0.jar:2.14.0]
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
[java.base:]
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
[java.base:]
at
org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
[artemis-commons-2.14.0.jar:2.14.0]
## And then I got about 1 billion of these:
Page 11, message 2,232 could not be found on offset 2,894,573, with
starting message 2,232. This represents a logic error or inconsistency on
the data, and the system will try once again from the beggining of the page
file.
Page file 000000011.page had incomplete records at position 2,895,805 at
record number 2,233
## Followed by an equally disturbing number of these:
Destination ActiveMQ.Advisory.TempQueue has an inconsistent and negative
address size=-13,632.
After restarting things looked OK, but heap utilization is still trending
upward. I also see more of the "Destination ActiveMQ.Advisory.TempQueue has
an inconsistent and negative address size=-13,632." errors appearing. I
also see where connections are timing out and being cleaned up, but the
overall connection count looks good -- between 26 and 28.
What does this indicate?
