daves created ARTEMIS-4307: ------------------------------ Summary: AMQP ARtemis "hangs" on single message delivery Key: ARTEMIS-4307 URL: https://issues.apache.org/jira/browse/ARTEMIS-4307 Project: ActiveMQ Artemis Issue Type: Bug Components: AMQP Affects Versions: 2.28.0 Reporter: daves
My Setup: - Multiple clients ~500 send messages to a queue in Artemis using AMQP - A single application reads consumes the messages from the queue using AMQP At some point - I sadly don't know how to reproduce it - my consuming client does not receive messages anymore. >From the perspective of my client the call to receiving new messages (AMQP.NET >lite) just hands/waits for messages and never returns. Also, It still looks >like the connection is open and healthy. On the Artemis side I see two different pictures. - In the Broker console I can see a consumer from my client, and it tells me that 1 Messages is in Transit. ...as if Artemis is waiting for an ACK for the message. In the Artemis log is see this: {code:java} 2023-06-09 15:11:37,934 WARN [org.apache.activemq.artemis.protocol.amqp.proton.ProtonServerSenderContext] Array must not be empty or null java.lang.IllegalArgumentException: Array must not be empty or null at org.apache.qpid.proton.codec.CompositeReadableBuffer.append(CompositeReadableBuffer.java:691) ~[proton-j-0.34.0.jar:?] at org.apache.qpid.proton.engine.impl.DeliveryImpl.send(DeliveryImpl.java:345) ~[proton-j-0.34.0.jar:?] at org.apache.qpid.proton.engine.impl.SenderImpl.send(SenderImpl.java:74) ~[proton-j-0.34.0.jar:?] at org.apache.activemq.artemis.protocol.amqp.proton.ProtonServerSenderContext$LargeMessageDeliveryContext.deliverInitialPacket(ProtonServerSenderContext.java:715) ~[artemis-amqp-protocol-2.28.0.jar:2.28.0] at org.apache.activemq.artemis.protocol.amqp.proton.ProtonServerSenderContext$LargeMessageDeliveryContext.deliver(ProtonServerSenderContext.java:622) ~[artemis-amqp-protocol-2.28.0.jar:2.28.0] at org.apache.activemq.artemis.protocol.amqp.proton.ProtonServerSenderContext.deliverLarge(ProtonServerSenderContext.java:837) ~[artemis-amqp-protocol-2.28.0.jar:2.28.0] at org.apache.activemq.artemis.protocol.amqp.proton.ProtonServerSenderContext.executeDelivery(ProtonServerSenderContext.java:567) ~[artemis-amqp-protocol-2.28.0.jar:2.28.0] at org.apache.activemq.artemis.core.paging.cursor.PagedReferenceImpl.run(PagedReferenceImpl.java:116) ~[artemis-server-2.28.0.jar:2.28.0] at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:174) ~[netty-common-4.1.86.Final.jar:4.1.86.Final] at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:167) ~[netty-common-4.1.86.Final.jar:4.1.86.Final] at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470) ~[netty-common-4.1.86.Final.jar:4.1.86.Final] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:566) ~[netty-transport-4.1.86.Final.jar:4.1.86.Final] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) ~[netty-common-4.1.86.Final.jar:4.1.86.Final] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[netty-common-4.1.86.Final.jar:4.1.86.Final] at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118) ~[artemis-commons-2.28.0.jar:?] 2023-06-09 15:11:37,934 WARN [org.apache.activemq.artemis.core.server] AMQ222151: removing consumer which did not handle a message, consumer=ServerConsumerImpl [id=3, filter=null, binding=LocalQueueBinding [address=MyQName, queue=QueueImpl[name=MyQName, postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=MyApp], temp=false]@776d381a, filter=null, name=MyQName, clusterName=MyQName431ed7e1-05dd-11ee-ade9-f44d30e2ecf9]], message=PagedReferenceImpl [message=PagedMessageImpl [queueIDs=[47], transactionID=-1, page=25, message=AMQPLargeMessage( [durable=false, messageID=12940183, address=MyQName, size=0, scanningStatus=SCANNED, applicationProperties={atmid=CHE80000128, content-type=application/x-protobuf, content-version=2.0, process-id=FireAndForgetJob_AtmStatusRequest_CHE80000128_ConfigurationRequest_f2eb7a14-fc1e-4439-8c83-4e836e83c818, content-status=200, Issued=2023-06-08 19:46:55.862 +02:00, x-version=7.9.44.348, AtmJobId=-1, AtmCommandTaskId=4, AtmCommandTaskResultId=-1, extensions=1, content-length=8145690}, messageAnnotations={}, properties=Properties{messageId=MyQName_2023-06-08T19:46:55.8712925+02:00_208f699f-9636-493f-9bfd-af4317e3dc45, userId=null, to='MyQName', subject='null', replyTo='null', correlationId=null, contentType=null, contentEncoding=null, absoluteExpiryTime=Fri Jun 07 19:46:55 CEST 2024, creationTime=null, groupId='null', groupSequence=null, replyToGroupId='null'}, extraProperties = TypedProperties[_AMQ_AD=MyQName]]], deliveryTime=0, persistedCount=0, deliveryCount=1, subscription=PageSubscriptionImpl [cursorId=47, queue=QueueImpl[name=MyQName, postOffice=PostOfficeImpl [server=ActiveMQServerImpl::name=MyApp], temp=false]@776d381a, filter = null]] java.lang.IllegalArgumentException: Array must not be empty or null at org.apache.qpid.proton.codec.CompositeReadableBuffer.append(CompositeReadableBuffer.java:691) ~[proton-j-0.34.0.jar:?] at org.apache.qpid.proton.engine.impl.DeliveryImpl.send(DeliveryImpl.java:345) ~[proton-j-0.34.0.jar:?] at org.apache.qpid.proton.engine.impl.SenderImpl.send(SenderImpl.java:74) ~[proton-j-0.34.0.jar:?] at org.apache.activemq.artemis.protocol.amqp.proton.ProtonServerSenderContext$LargeMessageDeliveryContext.deliverInitialPacket(ProtonServerSenderContext.java:715) ~[artemis-amqp-protocol-2.28.0.jar:2.28.0] at org.apache.activemq.artemis.protocol.amqp.proton.ProtonServerSenderContext$LargeMessageDeliveryContext.deliver(ProtonServerSenderContext.java:622) ~[artemis-amqp-protocol-2.28.0.jar:2.28.0] at org.apache.activemq.artemis.protocol.amqp.proton.ProtonServerSenderContext.deliverLarge(ProtonServerSenderContext.java:837) ~[artemis-amqp-protocol-2.28.0.jar:2.28.0] at org.apache.activemq.artemis.protocol.amqp.proton.ProtonServerSenderContext.executeDelivery(ProtonServerSenderContext.java:567) ~[artemis-amqp-protocol-2.28.0.jar:2.28.0] at org.apache.activemq.artemis.core.paging.cursor.PagedReferenceImpl.run(PagedReferenceImpl.java:116) ~[artemis-server-2.28.0.jar:2.28.0] at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:174) ~[netty-common-4.1.86.Final.jar:4.1.86.Final] at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:167) ~[netty-common-4.1.86.Final.jar:4.1.86.Final] at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470) ~[netty-common-4.1.86.Final.jar:4.1.86.Final] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:566) ~[netty-transport-4.1.86.Final.jar:4.1.86.Final] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) ~[netty-common-4.1.86.Final.jar:4.1.86.Final] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[netty-common-4.1.86.Final.jar:4.1.86.Final] at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118) ~[artemis-commons-2.28.0.jar:?] {code} To me it looks like Artemis tries to close the session/connection of my consuming client but fails to do so. Restarting Artemis or restarting my consumer does not change anything. So as soon as the problem occurs it does not go away anymore unless a delete the messages in the queue. So far I've no way to force/reproduce this problem, it just sporadically occurs. -- This message was sent by Atlassian Jira (v8.20.10#820010)