[ https://issues.apache.org/jira/browse/ARTEMIS-2261?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Keith Wall updated ARTEMIS-2261: -------------------------------- Description: Use case: {{global-max-size}} is set to some limit to prevent the broker from falling over. The broker is configured with N queues which are all blocked by this limit. If this limit is reached and you attempt to control the Broker with Management (over AMQP), whilst the management requests now processed by the Broker (owing to the work of ARTEMIS-1710), the management responses are lost if the response queue was created as a dynamic node. This is because the dynamically created node's name means that management response is not special cased are in stored. One can workaround the problem by establishing (non temporary) response queues that have a name prefixed with {{activemq.management}} e.g. {{activemq.management'.reply.<<uuid>>}} but this gives the application more work to do to ensure the response queues are lifecycle properly. {{ 2019-02-19T20:38:43.356Z WARN [ProtonServerReceiverContext] AMQ229102: Address "54111434-5686-4dbb-8a94-dd0182dbe7eb" is full.: ActiveMQAddressFullException[errorType=ADDRESS_FULL message=AMQ229102: Address "54111434-5686-4dbb-8a94-dd0182dbe7eb" is full.]}} {{ at org.apache.activemq.artemis.core.paging.impl.PagingStoreImpl.page(PagingStoreImpl.java:780)}} {{ at org.apache.activemq.artemis.core.persistence.impl.journal.AbstractJournalStorageManager.addToPage(AbstractJournalStorageManager.java:2019)}} {{ at org.apache.activemq.artemis.core.postoffice.impl.PostOfficeImpl.processRoute(PostOfficeImpl.java:1203)}} {{ at org.apache.activemq.artemis.core.postoffice.impl.PostOfficeImpl.route(PostOfficeImpl.java:904)}} {{ at org.apache.activemq.artemis.core.postoffice.impl.PostOfficeImpl.route(PostOfficeImpl.java:799)}} {{ at org.apache.activemq.artemis.core.server.impl.ServerSessionImpl.doSend(ServerSessionImpl.java:1801)}} {{ at org.apache.activemq.artemis.core.server.impl.ServerSessionImpl.handleManagementMessage(ServerSessionImpl.java:1698)}} {{ at org.apache.activemq.artemis.core.server.impl.ServerSessionImpl.send(ServerSessionImpl.java:1458)}} {{ at org.apache.activemq.artemis.protocol.amqp.broker.AMQPSessionCallback.serverSend(AMQPSessionCallback.java:550)}} {{ at org.apache.activemq.artemis.protocol.amqp.broker.AMQPSessionCallback.serverSend(AMQPSessionCallback.java:509)}} {{ at org.apache.activemq.artemis.protocol.amqp.proton.ProtonServerReceiverContext.onMessage(ProtonServerReceiverContext.java:284)}} {{ at org.apache.activemq.artemis.protocol.amqp.proton.AMQPConnectionContext.onDelivery(AMQPConnectionContext.java:519)}} {{ at org.apache.activemq.artemis.protocol.amqp.proton.handler.Events.dispatch(Events.java:92)}} {{ at org.apache.activemq.artemis.protocol.amqp.proton.handler.ProtonHandler.dispatch(ProtonHandler.java:479)}} {{ at org.apache.activemq.artemis.protocol.amqp.proton.handler.ProtonHandler.flush(ProtonHandler.java:292)}} {{ at org.apache.activemq.artemis.protocol.amqp.proton.handler.ProtonHandler.inputBuffer(ProtonHandler.java:257)}} {{ at org.apache.activemq.artemis.protocol.amqp.proton.AMQPConnectionContext.inputBuffer(AMQPConnectionContext.java:158)}} {{ at org.apache.activemq.artemis.protocol.amqp.broker.ActiveMQProtonRemotingConnection.bufferReceived(ActiveMQProtonRemotingConnection.java:148)}} {{ at org.apache.activemq.artemis.core.remoting.server.impl.RemotingServiceImpl$DelegatingBufferHandler.bufferReceived(RemotingServiceImpl.java:643)}} {{ at org.apache.activemq.artemis.core.remoting.impl.netty.ActiveMQChannelHandler.channelRead(ActiveMQChannelHandler.java:73)}} {{ at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)}} {{ at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)}} {{ at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)}} {{ at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1407)}} {{ at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1177)}} {{ at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1221)}} {{ at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489)}} {{ at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428)}} {{ at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265)}} {{ at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)}} {{ at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)}} {{ at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)}} {{ at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1434)}} {{ at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)}} {{ at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)}} {{ at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:965)}} {{ at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:808)}} {{ at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:404)}} {{ at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:304)}} {{ at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884)}} {{ at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)}} was: Use case: {{global-max-size}} is set to some limit to prevent the broker from falling over. The broker is configured with N queues which are all blocked by this limit. If this limit is reached and you attempt to control the Broker with Management (over AMQP), whilst the management requests now reach the Broker (no longer rejected owing to the work of ARTEMIS-1710), the management responses are lost if the response queue was created as a dynamic node. {{2019-02-19T20:38:43.356Z WARN [ProtonServerReceiverContext] AMQ229102: Address "54111434-5686-4dbb-8a94-dd0182dbe7eb" is full.: ActiveMQAddressFullException[errorType=ADDRESS_FULL message=AMQ229102: Address "54111434-5686-4dbb-8a94-dd0182dbe7eb" is full.] at org.apache.activemq.artemis.core.paging.impl.PagingStoreImpl.page(PagingStoreImpl.java:780) at org.apache.activemq.artemis.core.persistence.impl.journal.AbstractJournalStorageManager.addToPage(AbstractJournalStorageManager.java:2019) at org.apache.activemq.artemis.core.postoffice.impl.PostOfficeImpl.processRoute(PostOfficeImpl.java:1203) at org.apache.activemq.artemis.core.postoffice.impl.PostOfficeImpl.route(PostOfficeImpl.java:904) at org.apache.activemq.artemis.core.postoffice.impl.PostOfficeImpl.route(PostOfficeImpl.java:799) at org.apache.activemq.artemis.core.server.impl.ServerSessionImpl.doSend(ServerSessionImpl.java:1801) at org.apache.activemq.artemis.core.server.impl.ServerSessionImpl.handleManagementMessage(ServerSessionImpl.java:1698) at org.apache.activemq.artemis.core.server.impl.ServerSessionImpl.send(ServerSessionImpl.java:1458) at org.apache.activemq.artemis.protocol.amqp.broker.AMQPSessionCallback.serverSend(AMQPSessionCallback.java:550) at org.apache.activemq.artemis.protocol.amqp.broker.AMQPSessionCallback.serverSend(AMQPSessionCallback.java:509) at org.apache.activemq.artemis.protocol.amqp.proton.ProtonServerReceiverContext.onMessage(ProtonServerReceiverContext.java:284) at org.apache.activemq.artemis.protocol.amqp.proton.AMQPConnectionContext.onDelivery(AMQPConnectionContext.java:519) at org.apache.activemq.artemis.protocol.amqp.proton.handler.Events.dispatch(Events.java:92) at org.apache.activemq.artemis.protocol.amqp.proton.handler.ProtonHandler.dispatch(ProtonHandler.java:479) at org.apache.activemq.artemis.protocol.amqp.proton.handler.ProtonHandler.flush(ProtonHandler.java:292) at org.apache.activemq.artemis.protocol.amqp.proton.handler.ProtonHandler.inputBuffer(ProtonHandler.java:257) at org.apache.activemq.artemis.protocol.amqp.proton.AMQPConnectionContext.inputBuffer(AMQPConnectionContext.java:158) at org.apache.activemq.artemis.protocol.amqp.broker.ActiveMQProtonRemotingConnection.bufferReceived(ActiveMQProtonRemotingConnection.java:148) at org.apache.activemq.artemis.core.remoting.server.impl.RemotingServiceImpl$DelegatingBufferHandler.bufferReceived(RemotingServiceImpl.java:643) at org.apache.activemq.artemis.core.remoting.impl.netty.ActiveMQChannelHandler.channelRead(ActiveMQChannelHandler.java:73) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1407) at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1177) at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1221) at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489) at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1434) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:965) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:808) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:404) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:304) at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884) at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118) }} > Management response messages are lost when Broker exceeds global-max-size > limit and dynamic response queues are in use > ---------------------------------------------------------------------------------------------------------------------- > > Key: ARTEMIS-2261 > URL: https://issues.apache.org/jira/browse/ARTEMIS-2261 > Project: ActiveMQ Artemis > Issue Type: Bug > Components: AMQP > Reporter: Keith Wall > Priority: Major > > Use case: {{global-max-size}} is set to some limit to prevent the broker from > falling over. The broker is configured with N queues which are all blocked by > this limit. > If this limit is reached and you attempt to control the Broker with > Management (over AMQP), whilst the management requests now processed by the > Broker (owing to the work of ARTEMIS-1710), the management responses are lost > if the response queue was created as a dynamic node. This is because the > dynamically created node's name means that management response is not special > cased are in stored. > One can workaround the problem by establishing (non temporary) response > queues that have a name prefixed with {{activemq.management}} e.g. > {{activemq.management'.reply.<<uuid>>}} but this gives the application more > work to do to ensure the response queues are lifecycle properly. > {{ 2019-02-19T20:38:43.356Z WARN [ProtonServerReceiverContext] AMQ229102: > Address "54111434-5686-4dbb-8a94-dd0182dbe7eb" is full.: > ActiveMQAddressFullException[errorType=ADDRESS_FULL message=AMQ229102: > Address "54111434-5686-4dbb-8a94-dd0182dbe7eb" is full.]}} > {{ at > org.apache.activemq.artemis.core.paging.impl.PagingStoreImpl.page(PagingStoreImpl.java:780)}} > {{ at > org.apache.activemq.artemis.core.persistence.impl.journal.AbstractJournalStorageManager.addToPage(AbstractJournalStorageManager.java:2019)}} > {{ at > org.apache.activemq.artemis.core.postoffice.impl.PostOfficeImpl.processRoute(PostOfficeImpl.java:1203)}} > {{ at > org.apache.activemq.artemis.core.postoffice.impl.PostOfficeImpl.route(PostOfficeImpl.java:904)}} > {{ at > org.apache.activemq.artemis.core.postoffice.impl.PostOfficeImpl.route(PostOfficeImpl.java:799)}} > {{ at > org.apache.activemq.artemis.core.server.impl.ServerSessionImpl.doSend(ServerSessionImpl.java:1801)}} > {{ at > org.apache.activemq.artemis.core.server.impl.ServerSessionImpl.handleManagementMessage(ServerSessionImpl.java:1698)}} > {{ at > org.apache.activemq.artemis.core.server.impl.ServerSessionImpl.send(ServerSessionImpl.java:1458)}} > {{ at > org.apache.activemq.artemis.protocol.amqp.broker.AMQPSessionCallback.serverSend(AMQPSessionCallback.java:550)}} > {{ at > org.apache.activemq.artemis.protocol.amqp.broker.AMQPSessionCallback.serverSend(AMQPSessionCallback.java:509)}} > {{ at > org.apache.activemq.artemis.protocol.amqp.proton.ProtonServerReceiverContext.onMessage(ProtonServerReceiverContext.java:284)}} > {{ at > org.apache.activemq.artemis.protocol.amqp.proton.AMQPConnectionContext.onDelivery(AMQPConnectionContext.java:519)}} > {{ at > org.apache.activemq.artemis.protocol.amqp.proton.handler.Events.dispatch(Events.java:92)}} > {{ at > org.apache.activemq.artemis.protocol.amqp.proton.handler.ProtonHandler.dispatch(ProtonHandler.java:479)}} > {{ at > org.apache.activemq.artemis.protocol.amqp.proton.handler.ProtonHandler.flush(ProtonHandler.java:292)}} > {{ at > org.apache.activemq.artemis.protocol.amqp.proton.handler.ProtonHandler.inputBuffer(ProtonHandler.java:257)}} > {{ at > org.apache.activemq.artemis.protocol.amqp.proton.AMQPConnectionContext.inputBuffer(AMQPConnectionContext.java:158)}} > {{ at > org.apache.activemq.artemis.protocol.amqp.broker.ActiveMQProtonRemotingConnection.bufferReceived(ActiveMQProtonRemotingConnection.java:148)}} > {{ at > org.apache.activemq.artemis.core.remoting.server.impl.RemotingServiceImpl$DelegatingBufferHandler.bufferReceived(RemotingServiceImpl.java:643)}} > {{ at > org.apache.activemq.artemis.core.remoting.impl.netty.ActiveMQChannelHandler.channelRead(ActiveMQChannelHandler.java:73)}} > {{ at > io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)}} > {{ at > io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)}} > {{ at > io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)}} > {{ at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1407)}} > {{ at > io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1177)}} > {{ at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1221)}} > {{ at > io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489)}} > {{ at > io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428)}} > {{ at > io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265)}} > {{ at > io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)}} > {{ at > io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)}} > {{ at > io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)}} > {{ at > io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1434)}} > {{ at > io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)}} > {{ at > io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)}} > {{ at > io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:965)}} > {{ at > io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:808)}} > {{ at > io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:404)}} > {{ at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:304)}} > {{ at > io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884)}} > {{ at > org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)}} -- This message was sent by Atlassian JIRA (v7.6.3#76005)