[
https://issues.apache.org/jira/browse/ARTEMIS-4217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17816194#comment-17816194
]
daves commented on ARTEMIS-4217:
--------------------------------
Hi [~jbertram]
I planned to upload the file together with a comment, but I got interrupted
while writing the comment, sorry.
I did some more testing and found a way to reproduce the problem rellably.
For the problem to occure the system running the broker must be in a low memory
state. I've added a small project to the issue to reproduce the problem.
Setup:
Broker: The problem exists since version 2.28 I used 2.31.2 for this test.
2.27.1 is the last version in which the problem never occurs.
Broker config:
{code:xml}
<?xml version="1.0"?>
<configuration xmlns="urn:activemq"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance";
xmlns:xi="http://www.w3.org/2001/XInclude"; xsi:schemaLocation="urn:activemq
/schema/artemis-configuration.xsd">
<core xmlns="urn:activemq:core"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance";
xsi:schemaLocation="urn:activemq:core ">
<name>Argos</name>
<persistence-enabled>true</persistence-enabled>
<journal-type>NIO</journal-type>
<paging-directory>C:/Logisoft/Data/Artemis/Argos/data/paging</paging-directory>
<bindings-directory>C:/Logisoft/Data/Artemis/Argos/data/bindings</bindings-directory>
<journal-directory>C:/Logisoft/Data/Artemis/Argos/data/journal</journal-directory>
<large-messages-directory>C:/Logisoft/Data/Artemis/Argos/data/large-messages</large-messages-directory>
<journal-datasync>true</journal-datasync>
<journal-min-files>2</journal-min-files>
<journal-pool-files>10</journal-pool-files>
<journal-device-block-size>4096</journal-device-block-size>
<journal-file-size>10M</journal-file-size>
<journal-buffer-timeout>1404000</journal-buffer-timeout>
<journal-max-io>1</journal-max-io>
<disk-scan-period>5000</disk-scan-period>
<max-disk-usage>99</max-disk-usage>
<critical-analyzer>true</critical-analyzer>
<critical-analyzer-timeout>120000</critical-analyzer-timeout>
<critical-analyzer-check-period>60000</critical-analyzer-check-period>
<critical-analyzer-policy>HALT</critical-analyzer-policy>
<page-sync-timeout>1196000</page-sync-timeout>
<acceptors>
<acceptor
name="amqp">tcp://0.0.0.0:5672?tcpSendBufferSize=20000000;tcpReceiveBufferSize=20000000;minLargeMessageSize=5000000;amqpMinLargeMessageSize=5000000;consumerWindowSize=10485760;directDeliver=false;protocols=AMQP;useEpoll=true;amqpCredits=1000;amqpMinCredits=300</acceptor>
</acceptors>
<security-settings>
<security-setting match="#">
<permission type="createNonDurableQueue" roles="amq" />
<permission type="deleteNonDurableQueue" roles="amq" />
<permission type="createDurableQueue" roles="amq" />
<permission type="deleteDurableQueue" roles="amq" />
<permission type="createAddress" roles="amq" />
<permission type="deleteAddress" roles="amq" />
<permission type="consume" roles="amq" />
<permission type="browse" roles="amq" />
<permission type="send" roles="amq" />
<permission type="manage" roles="amq" />
</security-setting>
</security-settings>
<address-settings>
<address-setting match="activemq.management#">
<redelivery-delay>0</redelivery-delay>
<max-size-bytes>-1</max-size-bytes>
<message-counter-history-day-limit>10</message-counter-history-day-limit>
<address-full-policy>PAGE</address-full-policy>
<auto-create-queues>true</auto-create-queues>
<auto-create-addresses>true</auto-create-addresses>
<auto-create-jms-queues>true</auto-create-jms-queues>
<auto-create-jms-topics>true</auto-create-jms-topics>
</address-setting>
<address-setting match="#">
<redelivery-delay-multiplier>1.5</redelivery-delay-multiplier>
<redelivery-delay>5000</redelivery-delay>
<max-redelivery-delay>60000</max-redelivery-delay>
<max-delivery-attempts>10</max-delivery-attempts>
<page-size-bytes>10485760</page-size-bytes>
<max-size-bytes>104857600</max-size-bytes>
<page-max-cache-size>2</page-max-cache-size>
<address-full-policy>PAGE</address-full-policy>
<message-counter-history-day-limit>10</message-counter-history-day-limit>
<auto-create-queues>true</auto-create-queues>
<auto-create-addresses>true</auto-create-addresses>
<auto-create-jms-queues>true</auto-create-jms-queues>
<auto-create-jms-topics>true</auto-create-jms-topics>
<auto-delete-jms-queues>false</auto-delete-jms-queues>
<auto-delete-created-queues>false</auto-delete-created-queues>
<auto-delete-queues>false</auto-delete-queues>
<auto-delete-addresses>false</auto-delete-addresses>
</address-setting>
</address-settings>
<addresses>
<address name="DLQ">
<anycast>
<queue name="DLQ" />
</anycast>
</address>
<address name="ExpiryQueue">
<anycast>
<queue name="ExpiryQueue" />
</anycast>
</address>
</addresses>
</core>
</configuration>
{code}
Repro tool:
Extract ArtemisReproMemory.zip and run the project.
It is a .net application and should run on all major platforms, but I tested it
only on Windows (11) . To build and run it, you need the .net8.0 SDK from here:
https://dotnet.microsoft.com/en-us/download/visual-studio-sdks
Run this command to build the project (in the root of the extracted directory):
{code:cmd}
dotnet build -c Release
{code}
Start the built application like this:
{code:cmd}
cd .\ArtemisReproMemory\bin\Release\net8.0\
dotnet .\ArtemisReproMemory.dll -a amqp://admin:password@localhost:5672 -b 10
{code}
Use the -a switch to specify a broker URL (AMQP)
Use -b to allocate the specified number of bytes to simulate a high memory
state. (Only makes sense if the broker is running on the same machine)
The repro tool will send large messages to the specified broker using the AMQP
protocol and simultaneously consume the messages. The actions of the repro tool
trigger the problem (
Failed to convert message. Sending it to Dead Letter Address.) on all machines
I tested with.
In some cases, it took considerably longer for the problem to occur, if the
broker was started before the system memory was all "in use". So it can help to
start the broker after the repro tool is up and running (using the -b flag).
Artemis will, using this setup, generate the following log entries and loose
messages:
{code:java}
2024-02-09 10:30:17,320 WARN [org.apache.activemq.artemis.core.server]
AMQ222150: Sending message
Reference[8590002935]:NON-RELIABLE:LargeServerMessage[messageID=8590002935,durable=false,userID=d5f605da-c72d-11ee-891c-a059505d02d7,priority=4,
timestamp=0,expiration=0, durable=false, address=test-q,
properties=TypedProperties[NATIVE_MESSAGE_ID=ID:AMQP_NO_PREFIX:test-producer
#18, JMS_AMQP_ORIGINAL_ENCODING=6, _AMQ_LARGE_SIZE=1048576]]@964273977 to Dead
Letter Address, but there is no Dead Letter Address configured for queue test-q
so dropping it
2024-02-09 10:30:29,752 WARN
[org.apache.activemq.artemis.protocol.amqp.logger] AMQ111005: Failed to convert
message. Sending it to Dead Letter Address.
org.apache.activemq.artemis.protocol.amqp.converter.coreWrapper.ConversionException:
java.nio.channels.AsynchronousCloseException
at
org.apache.activemq.artemis.protocol.amqp.converter.CoreAmqpConverter.fromCore(CoreAmqpConverter.java:318)
~[artemis-amqp-protocol-2.31.2.jar:2.31.2]
at
org.apache.activemq.artemis.protocol.amqp.converter.CoreAmqpConverter.checkAMQP(CoreAmqpConverter.java:79)
~[artemis-amqp-protocol-2.31.2.jar:2.31.2]
at
org.apache.activemq.artemis.protocol.amqp.proton.ProtonServerSenderContext.executeDelivery(ProtonServerSenderContext.java:572)
~[artemis-amqp-protocol-2.31.2.jar:2.31.2]
at
org.apache.activemq.artemis.core.server.impl.MessageReferenceImpl.run(MessageReferenceImpl.java:131)
[artemis-server-2.31.2.jar:2.31.2]
at
io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:173)
[netty-common-4.1.100.Final.jar:4.1.100.Final]
at
io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:166)
[netty-common-4.1.100.Final.jar:4.1.100.Final]
at
io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470)
[netty-common-4.1.100.Final.jar:4.1.100.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:569)
[netty-transport-4.1.100.Final.jar:4.1.100.Final]
at
io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
[netty-common-4.1.100.Final.jar:4.1.100.Final]
at
io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
[netty-common-4.1.100.Final.jar:4.1.100.Final]
at
org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
[artemis-commons-2.31.2.jar:2.31.2]
Caused by: java.lang.RuntimeException:
java.nio.channels.AsynchronousCloseException
at
org.apache.activemq.artemis.core.persistence.impl.journal.LargeBody.getBodyBufferSize(LargeBody.java:299)
~[artemis-server-2.31.2.jar:2.31.2]
at
org.apache.activemq.artemis.core.persistence.impl.journal.LargeServerMessageImpl.getBodyBufferSize(LargeServerMessageImpl.java:256)
~[artemis-server-2.31.2.jar:2.31.2]
at
org.apache.activemq.artemis.protocol.amqp.converter.coreWrapper.CoreBytesMessageWrapper.getBodyLength(CoreBytesMessageWrapper.java:98)
~[artemis-amqp-protocol-2.31.2.jar:2.31.2]
at
org.apache.activemq.artemis.protocol.amqp.converter.coreWrapper.CoreBytesMessageWrapper.getBinaryFromMessageBody(CoreBytesMessageWrapper.java:68)
~[artemis-amqp-protocol-2.31.2.jar:2.31.2]
at
org.apache.activemq.artemis.protocol.amqp.converter.coreWrapper.CoreBytesMessageWrapper.createAMQPSection(CoreBytesMessageWrapper.java:78)
~[artemis-amqp-protocol-2.31.2.jar:2.31.2]
at
org.apache.activemq.artemis.protocol.amqp.converter.CoreAmqpConverter.fromCore(CoreAmqpConverter.java:106)
~[artemis-amqp-protocol-2.31.2.jar:2.31.2]
... 10 more
Caused by: java.nio.channels.AsynchronousCloseException
at
java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:203)
~[?:?]
at sun.nio.ch.FileChannelImpl.endBlocking(FileChannelImpl.java:172)
~[?:?]
at sun.nio.ch.FileChannelImpl.size(FileChannelImpl.java:430) ~[?:?]
at
org.apache.activemq.artemis.core.io.nio.NIOSequentialFile.size(NIOSequentialFile.java:339)
~[artemis-journal-2.31.2.jar:2.31.2]
at
org.apache.activemq.artemis.core.persistence.impl.journal.LargeBody.getBodyBufferSize(LargeBody.java:291)
~[artemis-server-2.31.2.jar:2.31.2]
at
org.apache.activemq.artemis.core.persistence.impl.journal.LargeServerMessageImpl.getBodyBufferSize(LargeServerMessageImpl.java:256)
~[artemis-server-2.31.2.jar:2.31.2]
at
org.apache.activemq.artemis.protocol.amqp.converter.coreWrapper.CoreBytesMessageWrapper.getBodyLength(CoreBytesMessageWrapper.java:98)
~[artemis-amqp-protocol-2.31.2.jar:2.31.2]
at
org.apache.activemq.artemis.protocol.amqp.converter.coreWrapper.CoreBytesMessageWrapper.getBinaryFromMessageBody(CoreBytesMessageWrapper.java:68)
~[artemis-amqp-protocol-2.31.2.jar:2.31.2]
at
org.apache.activemq.artemis.protocol.amqp.converter.coreWrapper.CoreBytesMessageWrapper.createAMQPSection(CoreBytesMessageWrapper.java:78)
~[artemis-amqp-protocol-2.31.2.jar:2.31.2]
at
org.apache.activemq.artemis.protocol.amqp.converter.CoreAmqpConverter.fromCore(CoreAmqpConverter.java:106)
~[artemis-amqp-protocol-2.31.2.jar:2.31.2]
... 10 more
{code}
I hope this helps to better understand the problem.
> AMQ111005: Failed to convert message. Sending it to Dead Letter Address.
> ------------------------------------------------------------------------
>
> Key: ARTEMIS-4217
> URL: https://issues.apache.org/jira/browse/ARTEMIS-4217
> Project: ActiveMQ Artemis
> Issue Type: Bug
> Components: AMQP
> Affects Versions: 2.28.0
> Environment: Windows Server 2022 21H2
> openjdk 19.0.1 2022-10-18
> OpenJDK Runtime Environment (build 19.0.1+10-21)
> OpenJDK 64-Bit Server VM (build 19.0.1+10-21, mixed mode, sharing)
> Reporter: daves
> Priority: Major
> Attachments: ArtemisConvertError.zip, ArtemisReproMemory.zip,
> image-2023-06-16-15-59-25-689.png, image-2023-06-16-15-59-25-721.png
>
>
> Some of the AMQP messages sent by my client never arrive at the consumer. In
> the Artemis log I found the following exception:
> {noformat}
> 2023-03-23 18:06:58,084 WARN
> [org.apache.activemq.artemis.protocol.amqp.logger] AMQ111005: Failed to
> convert message. Sending it to Dead Letter Address.
> org.apache.activemq.artemis.protocol.amqp.converter.coreWrapper.ConversionException:
> java.nio.channels.ClosedChannelException
> at
> org.apache.activemq.artemis.protocol.amqp.converter.CoreAmqpConverter.fromCore(CoreAmqpConverter.java:318)
> ~[artemis-amqp-protocol-2.28.0.jar:2.28.0]
> at
> org.apache.activemq.artemis.protocol.amqp.converter.CoreAmqpConverter.checkAMQP(CoreAmqpConverter.java:79)
> ~[artemis-amqp-protocol-2.28.0.jar:2.28.0]
> at
> org.apache.activemq.artemis.protocol.amqp.proton.ProtonServerSenderContext.executeDelivery(ProtonServerSenderContext.java:561)
> ~[artemis-amqp-protocol-2.28.0.jar:2.28.0]
> at
> org.apache.activemq.artemis.core.server.impl.MessageReferenceImpl.run(MessageReferenceImpl.java:131)
> ~[artemis-server-2.28.0.jar:2.28.0]
> at
> io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:174)
> ~[netty-common-4.1.86.Final.jar:4.1.86.Final]
> at
> io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:167)
> ~[netty-common-4.1.86.Final.jar:4.1.86.Final]
> at
> io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470)
> ~[netty-common-4.1.86.Final.jar:4.1.86.Final]
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:569)
> ~[netty-transport-4.1.86.Final.jar:4.1.86.Final]
> at
> io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
> ~[netty-common-4.1.86.Final.jar:4.1.86.Final]
> at
> io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
> ~[netty-common-4.1.86.Final.jar:4.1.86.Final]
> at
> org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
> ~[artemis-commons-2.28.0.jar:?] Caused by: java.lang.RuntimeException:
> java.nio.channels.ClosedChannelException
> at
> org.apache.activemq.artemis.core.persistence.impl.journal.LargeBody.getBodyBufferSize(LargeBody.java:293)
> ~[artemis-server-2.28.0.jar:2.28.0]
> at
> org.apache.activemq.artemis.core.persistence.impl.journal.LargeServerMessageImpl.getBodyBufferSize(LargeServerMessageImpl.java:263)
> ~[artemis-server-2.28.0.jar:2.28.0]
> at
> org.apache.activemq.artemis.protocol.amqp.converter.coreWrapper.CoreBytesMessageWrapper.getBodyLength(CoreBytesMessageWrapper.java:98)
> ~[artemis-amqp-protocol-2.28.0.jar:2.28.0]
> at
> org.apache.activemq.artemis.protocol.amqp.converter.coreWrapper.CoreBytesMessageWrapper.getBinaryFromMessageBody(CoreBytesMessageWrapper.java:68)
> ~[artemis-amqp-protocol-2.28.0.jar:2.28.0]
> at
> org.apache.activemq.artemis.protocol.amqp.converter.coreWrapper.CoreBytesMessageWrapper.createAMQPSection(CoreBytesMessageWrapper.java:78)
> ~[artemis-amqp-protocol-2.28.0.jar:2.28.0]
> at
> org.apache.activemq.artemis.protocol.amqp.converter.CoreAmqpConverter.fromCore(CoreAmqpConverter.java:106)
> ~[artemis-amqp-protocol-2.28.0.jar:2.28.0]
> ... 10 more
> Caused by: java.nio.channels.ClosedChannelException
> at sun.nio.ch.FileChannelImpl.ensureOpen(FileChannelImpl.java:165)
> ~[?:?]
> at sun.nio.ch.FileChannelImpl.size(FileChannelImpl.java:416) ~[?:?]
> at
> org.apache.activemq.artemis.core.io.nio.NIOSequentialFile.size(NIOSequentialFile.java:339)
> ~[artemis-journal-2.28.0.jar:2.28.0]
> at
> org.apache.activemq.artemis.core.persistence.impl.journal.LargeBody.getBodyBufferSize(LargeBody.java:285)
> ~[artemis-server-2.28.0.jar:2.28.0]
> at
> org.apache.activemq.artemis.core.persistence.impl.journal.LargeServerMessageImpl.getBodyBufferSize(LargeServerMessageImpl.java:263)
> ~[artemis-server-2.28.0.jar:2.28.0]
> at
> org.apache.activemq.artemis.protocol.amqp.converter.coreWrapper.CoreBytesMessageWrapper.getBodyLength(CoreBytesMessageWrapper.java:98)
> ~[artemis-amqp-protocol-2.28.0.jar:2.28.0]
> at
> org.apache.activemq.artemis.protocol.amqp.converter.coreWrapper.CoreBytesMessageWrapper.getBinaryFromMessageBody(CoreBytesMessageWrapper.java:68)
> ~[artemis-amqp-protocol-2.28.0.jar:2.28.0]
> at
> org.apache.activemq.artemis.protocol.amqp.converter.coreWrapper.CoreBytesMessageWrapper.createAMQPSection(CoreBytesMessageWrapper.java:78)
> ~[artemis-amqp-protocol-2.28.0.jar:2.28.0]
> at
> org.apache.activemq.artemis.protocol.amqp.converter.CoreAmqpConverter.fromCore(CoreAmqpConverter.java:106)
> ~[artemis-amqp-protocol-2.28.0.jar:2.28.0]
> ... 10 more{noformat}
> I tried to reproduce the error with trace log level enabled, but the problem
> never occurs with trace enabled (same message payload). Maybe it is a timing
> issue which not occurs when Artemis processes messages much slower due to
> additional tracing...?
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
