I will give that a shot. Thanks. But the reason I think something is wrong is that once the flooding starts, it doesn't stop until I restart the activemq service on one of the machines. The flooding occurs even if the session that created the queue is long since gone. It only takes one request-reply message to start this seeming loop between "add destination" and "remove destination". You can see that in the logs I've attached previously. It completely pegs the CPU on one machine pretty much crippling it. Mind you, at this point there are no producers and only consumers listening for new messages. The only way to stop this endless loop of messages is to restart the activemq service.
I've been trying to reproduce it in different toy environments setup to be like things are in production and have been unsuccessful. I've seen the advisories going back and forth as they are logged but they don't loop endlessly like I'm seeing in production. I'm completely okay with the extra traffic caused by advisories, just as long as there isn't a loop :) These two machines I've spoken of talk to each other over VPN, which I have yet to test. On Mon, May 14, 2012 at 4:19 PM, Gary Tully <gary.tu...@gmail.com> wrote: > this is expected, but the flooding impact should be minimal with a > small number of brokers in the network and a reasonable number of > connections. > > each network bridge will register interest in all temp destination > advisories, so that it will be in a position to receive messages for > those destinations. It can't know in advance which messages may be > routed though it, so it needs to react to each advisory message and > create a local temp destination. When the connection that creates the > temp is closes, the temp destination gets closed and each broker > reacts to that event via the remove temp advisory. > > Temp destinations are often pooled for serial request reply messages > or multiplexed using selectors, to reduce the overhead of creation and > propagation, in a network scenario. The other option is to use regular > destinations for replys > > For temp destinations, there is an alternative solution that would > require a little more configuration but which would reduce the > overhead of unnecessary temp destination propagation. > You can disable auto temp destination propagation through the network > bridge, ie: bridgeTempDestinations=false. > Then you need to allow temp destination creation on the fly for a > producer, broker.allowTempAutoCreationOnSend=true > To finish, you need to configure > broker.schedulePeriodForDestinationPurge>0 to have the on the fly > created temp destinations purged when they are empty and enable it for > temp destinations. policy entry gcInactiveDestinations=true (this is > important because the auto created temp destination cannot be > associated with a connection in any sensible way) > > There is a test that exercises these options > > http://svn.apache.org/repos/asf/activemq/trunk/activemq-core/src/test/java/org/apache/activemq/usecases/RequestReplyNoAdvisoryNetworkTest.java > > > On 11 May 2012 17:45, Chris Robison <chrisdrobi...@gmail.com> wrote: > > Another interesting observation: as a test, I connected all producers and > > consumers to MSTMIP102 (the machine that doesn't have the network > > connector). I then started MSIPAP101 so that its network connector could > > start and connect to MSTMIP102. Request request-reply works fine between > > all producers and consumers, but after 1 or 2 request-replies, MSIPAP101 > is > > flooded with the temp queue advisories--even though there are no > producers > > or consumers connected to it. > > > > On Fri, May 11, 2012 at 10:09 AM, Chris Robison <chrisdrobi...@gmail.com > >wrote: > > > >> Here is the log from the latest run with the TTL down to 1. It still > >> floods after about 1 or 2 request-reply calls. It just seems to be > >> alternating between add temp queue and remove temp queue messages. > >> > >> On Fri, May 11, 2012 at 9:52 AM, Chris Robison <chrisdrobi...@gmail.com > >wrote: > >> > >>> Yes, because we have a spoke and hub topology where a client in one > spoke > >>> may need to request info from a client connected to a different spoke. > I > >>> suppose I could lower that value. > >>> > >>> > >>> On Fri, May 11, 2012 at 9:47 AM, Gary Tully <gary.tu...@gmail.com> > wrote: > >>> > >>>> Do you have a good reason to have networkTTL="5", if there is only one > >>>> broker, the default value of 1 will be fine. > >>>> > >>>> On 11 May 2012 16:32, Chris Robison <chrisdrobi...@gmail.com> wrote: > >>>> > If it helps, I've attached the configuration files for both > machines. > >>>> > > >>>> > > >>>> > On Fri, May 11, 2012 at 9:07 AM, Chris Robison < > >>>> chrisdrobi...@gmail.com> > >>>> > wrote: > >>>> >> > >>>> >> One thing I have noticed though is that when I restart the broker > on > >>>> the > >>>> >> other end, everything starts to work again until it floods again. > >>>> >> > >>>> >> > >>>> >> On Fri, May 11, 2012 at 9:05 AM, Chris Robison < > >>>> chrisdrobi...@gmail.com> > >>>> >> wrote: > >>>> >>> > >>>> >>> I'm on 5.6. And all brokers remain up and running. > >>>> >>> > >>>> >>> > >>>> >>> On Fri, May 11, 2012 at 8:48 AM, Gary Tully <gary.tu...@gmail.com > > > >>>> wrote: > >>>> >>>> > >>>> >>>> what version are you on? > >>>> >>>> Is there any chance that the broker at the other end of the > network > >>>> >>>> bridge is shutting down? > >>>> >>>> > >>>> >>>> On 11 May 2012 14:34, Chris Robison <chrisdrobi...@gmail.com> > >>>> wrote: > >>>> >>>> > I have a network of brokers and I am using a request-reply > system > >>>> >>>> > across > >>>> >>>> > that network, but I've noticed that after a few request-replys, > >>>> the > >>>> >>>> > broker > >>>> >>>> > network will get flooded with TempQueue advisory messages, then > >>>> some > >>>> >>>> > of the > >>>> >>>> > of the brokers will be reported as slow consumers and the > >>>> consumers > >>>> >>>> > won't > >>>> >>>> > receive messages after that. The advisory messages seem to be > >>>> repeated > >>>> >>>> > (e.g. add temp destination, remove temp destination) for the > same > >>>> temp > >>>> >>>> > destinations. Not quite sure how to handle this. Any > suggestions? > >>>> >>>> > > >>>> >>>> > Here are some of the errors are other things I'm seeing > >>>> repeatedly in > >>>> >>>> > logs > >>>> >>>> > when I enable debug. In fact, when this flood occurs, tons of > log > >>>> gets > >>>> >>>> > generated all at once. > >>>> >>>> > > >>>> >>>> > 2012-05-11 05:24:37,248 | DEBUG | Error occured while > processing > >>>> async > >>>> >>>> > command: DestinationInfo {commandId = 40578, responseRequired = > >>>> false, > >>>> >>>> > connectionId = ID:MSTMIP102-59588-1336713700699-3:2, > destination = > >>>> >>>> > temp-queue://ID:POTMIP102-54337-634722930065323786-1:18:1, > >>>> >>>> > operationType = > >>>> >>>> > 0, timeout = 0, brokerPath = > >>>> [ID:MSSD5D101-38580-1336708634042-0:1]}, > >>>> >>>> > exception: java.lang.IllegalStateException: Timer already > >>>> cancelled. | > >>>> >>>> > org.apache.activemq.broker.TransportConnection.Service | > ActiveMQ > >>>> >>>> > Transport: ssl:///10.60.252.28:54493 > >>>> >>>> > java.lang.IllegalStateException: Timer already cancelled. > >>>> >>>> > at java.util.Timer.sched(Timer.java:397) > >>>> >>>> > at java.util.Timer.schedule(Timer.java:248) > >>>> >>>> > at > >>>> >>>> > > >>>> >>>> > > >>>> > org.apache.activemq.thread.Scheduler.schedualPeriodically(Scheduler.java:49) > >>>> >>>> > at > org.apache.activemq.broker.region.Queue.start(Queue.java:889) > >>>> >>>> > at > >>>> >>>> > > >>>> >>>> > > >>>> > org.apache.activemq.broker.region.AbstractRegion.addDestination(AbstractRegion.java:143) > >>>> >>>> > at > >>>> >>>> > > >>>> >>>> > > >>>> > org.apache.activemq.broker.region.RegionBroker.addDestination(RegionBroker.java:289) > >>>> >>>> > at > >>>> >>>> > > >>>> >>>> > > >>>> > org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:145) > >>>> >>>> > at > >>>> >>>> > > >>>> >>>> > > >>>> > org.apache.activemq.advisory.AdvisoryBroker.addDestination(AdvisoryBroker.java:161) > >>>> >>>> > at > >>>> >>>> > > >>>> >>>> > > >>>> > org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:145) > >>>> >>>> > at > >>>> >>>> > > >>>> >>>> > > >>>> > org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:145) > >>>> >>>> > at > >>>> >>>> > > >>>> >>>> > > >>>> > org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:145) > >>>> >>>> > at > >>>> >>>> > > >>>> >>>> > > >>>> > org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:145) > >>>> >>>> > at > >>>> >>>> > > >>>> >>>> > > >>>> > org.apache.activemq.security.AuthorizationBroker.addDestination(AuthorizationBroker.java:81) > >>>> >>>> > at > >>>> >>>> > > >>>> >>>> > > >>>> > org.apache.activemq.security.AuthorizationBroker.addDestinationInfo(AuthorizationBroker.java:51) > >>>> >>>> > at > >>>> >>>> > > >>>> >>>> > > >>>> > org.apache.activemq.broker.MutableBrokerFilter.addDestinationInfo(MutableBrokerFilter.java:223) > >>>> >>>> > at > >>>> >>>> > > >>>> >>>> > > >>>> > org.apache.activemq.broker.TransportConnection.processAddDestination(TransportConnection.java:477) > >>>> >>>> > at > >>>> >>>> > > >>>> >>>> > > >>>> > org.apache.activemq.command.DestinationInfo.visit(DestinationInfo.java:122) > >>>> >>>> > at > >>>> >>>> > > >>>> >>>> > > >>>> > org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:292) > >>>> >>>> > at > >>>> >>>> > > >>>> >>>> > > >>>> > org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:150) > >>>> >>>> > at > >>>> >>>> > > >>>> >>>> > > >>>> > org.apache.activemq.transport.ResponseCorrelator.onCommand(ResponseCorrelator.java:116) > >>>> >>>> > at > >>>> >>>> > > >>>> >>>> > > >>>> > org.apache.activemq.transport.MutexTransport.onCommand(MutexTransport.java:50) > >>>> >>>> > at > >>>> >>>> > > >>>> >>>> > > >>>> > org.apache.activemq.transport.vm.VMTransport.doDispatch(VMTransport.java:135) > >>>> >>>> > at > >>>> >>>> > > >>>> >>>> > > >>>> > org.apache.activemq.transport.vm.VMTransport.dispatch(VMTransport.java:124) > >>>> >>>> > at > >>>> >>>> > > >>>> > org.apache.activemq.transport.vm.VMTransport.oneway(VMTransport.java:103) > >>>> >>>> > at > >>>> >>>> > > >>>> >>>> > > >>>> > org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:68) > >>>> >>>> > at > >>>> >>>> > > >>>> >>>> > > >>>> > org.apache.activemq.transport.ResponseCorrelator.oneway(ResponseCorrelator.java:60) > >>>> >>>> > at > >>>> >>>> > > >>>> >>>> > > >>>> > org.apache.activemq.network.DemandForwardingBridgeSupport.serviceRemoteConsumerAdvisory(DemandForwardingBridgeSupport.java:588) > >>>> >>>> > at > >>>> >>>> > > >>>> >>>> > > >>>> > org.apache.activemq.network.DemandForwardingBridgeSupport.serviceRemoteCommand(DemandForwardingBridgeSupport.java:420) > >>>> >>>> > at > >>>> >>>> > > >>>> >>>> > > >>>> > org.apache.activemq.network.DemandForwardingBridgeSupport$2.onCommand(DemandForwardingBridgeSupport.java:151) > >>>> >>>> > at > >>>> >>>> > > >>>> >>>> > > >>>> > org.apache.activemq.transport.ResponseCorrelator.onCommand(ResponseCorrelator.java:116) > >>>> >>>> > at > >>>> >>>> > > >>>> >>>> > > >>>> > org.apache.activemq.transport.MutexTransport.onCommand(MutexTransport.java:50) > >>>> >>>> > at > >>>> >>>> > > >>>> >>>> > > >>>> > org.apache.activemq.transport.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:113) > >>>> >>>> > at > >>>> >>>> > > >>>> >>>> > > >>>> > org.apache.activemq.transport.AbstractInactivityMonitor.onCommand(AbstractInactivityMonitor.java:229) > >>>> >>>> > at > >>>> >>>> > > >>>> >>>> > > >>>> > org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:83) > >>>> >>>> > at > >>>> >>>> > > >>>> >>>> > > >>>> > org.apache.activemq.transport.tcp.SslTransport.doConsume(SslTransport.java:91) > >>>> >>>> > at > >>>> >>>> > > >>>> >>>> > > >>>> > org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:222) > >>>> >>>> > at > >>>> >>>> > > >>>> > org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:204) > >>>> >>>> > > >>>> >>>> > > >>>> >>>> > > >>>> >>>> > 2012-05-11 05:24:37,248 | WARN | Async error occurred: > >>>> >>>> > java.lang.IllegalStateException: Timer already cancelled. | > >>>> >>>> > org.apache.activemq.broker.TransportConnection.Service | > ActiveMQ > >>>> >>>> > Transport: ssl:///10.60.252.28:54493 > >>>> >>>> > java.lang.IllegalStateException: Timer already cancelled. > >>>> >>>> > at java.util.Timer.sched(Timer.java:397) > >>>> >>>> > at java.util.Timer.schedule(Timer.java:248) > >>>> >>>> > at > >>>> >>>> > > >>>> >>>> > > >>>> > org.apache.activemq.thread.Scheduler.schedualPeriodically(Scheduler.java:49) > >>>> >>>> > at > org.apache.activemq.broker.region.Queue.start(Queue.java:889) > >>>> >>>> > at > >>>> >>>> > > >>>> >>>> > > >>>> > org.apache.activemq.broker.region.AbstractRegion.addDestination(AbstractRegion.java:143) > >>>> >>>> > at > >>>> >>>> > > >>>> >>>> > > >>>> > org.apache.activemq.broker.region.RegionBroker.addDestination(RegionBroker.java:289) > >>>> >>>> > at > >>>> >>>> > > >>>> >>>> > > >>>> > org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:145) > >>>> >>>> > at > >>>> >>>> > > >>>> >>>> > > >>>> > org.apache.activemq.advisory.AdvisoryBroker.addDestination(AdvisoryBroker.java:161) > >>>> >>>> > at > >>>> >>>> > > >>>> >>>> > > >>>> > org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:145) > >>>> >>>> > at > >>>> >>>> > > >>>> >>>> > > >>>> > org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:145) > >>>> >>>> > at > >>>> >>>> > > >>>> >>>> > > >>>> > org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:145) > >>>> >>>> > at > >>>> >>>> > > >>>> >>>> > > >>>> > org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:145) > >>>> >>>> > at > >>>> >>>> > > >>>> >>>> > > >>>> > org.apache.activemq.security.AuthorizationBroker.addDestination(AuthorizationBroker.java:81) > >>>> >>>> > at > >>>> >>>> > > >>>> >>>> > > >>>> > org.apache.activemq.security.AuthorizationBroker.addDestinationInfo(AuthorizationBroker.java:51) > >>>> >>>> > at > >>>> >>>> > > >>>> >>>> > > >>>> > org.apache.activemq.broker.MutableBrokerFilter.addDestinationInfo(MutableBrokerFilter.java:223) > >>>> >>>> > at > >>>> >>>> > > >>>> >>>> > > >>>> > org.apache.activemq.broker.TransportConnection.processAddDestination(TransportConnection.java:477) > >>>> >>>> > at > >>>> >>>> > > >>>> >>>> > > >>>> > org.apache.activemq.command.DestinationInfo.visit(DestinationInfo.java:122) > >>>> >>>> > at > >>>> >>>> > > >>>> >>>> > > >>>> > org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:292) > >>>> >>>> > at > >>>> >>>> > > >>>> >>>> > > >>>> > org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:150) > >>>> >>>> > at > >>>> >>>> > > >>>> >>>> > > >>>> > org.apache.activemq.transport.ResponseCorrelator.onCommand(ResponseCorrelator.java:116) > >>>> >>>> > at > >>>> >>>> > > >>>> >>>> > > >>>> > org.apache.activemq.transport.MutexTransport.onCommand(MutexTransport.java:50) > >>>> >>>> > at > >>>> >>>> > > >>>> >>>> > > >>>> > org.apache.activemq.transport.vm.VMTransport.doDispatch(VMTransport.java:135) > >>>> >>>> > at > >>>> >>>> > > >>>> >>>> > > >>>> > org.apache.activemq.transport.vm.VMTransport.dispatch(VMTransport.java:124) > >>>> >>>> > at > >>>> >>>> > > >>>> > org.apache.activemq.transport.vm.VMTransport.oneway(VMTransport.java:103) > >>>> >>>> > at > >>>> >>>> > > >>>> >>>> > > >>>> > org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:68) > >>>> >>>> > at > >>>> >>>> > > >>>> >>>> > > >>>> > org.apache.activemq.transport.ResponseCorrelator.oneway(ResponseCorrelator.java:60) > >>>> >>>> > at > >>>> >>>> > > >>>> >>>> > > >>>> > org.apache.activemq.network.DemandForwardingBridgeSupport.serviceRemoteConsumerAdvisory(DemandForwardingBridgeSupport.java:588) > >>>> >>>> > at > >>>> >>>> > > >>>> >>>> > > >>>> > org.apache.activemq.network.DemandForwardingBridgeSupport.serviceRemoteCommand(DemandForwardingBridgeSupport.java:420) > >>>> >>>> > at > >>>> >>>> > > >>>> >>>> > > >>>> > org.apache.activemq.network.DemandForwardingBridgeSupport$2.onCommand(DemandForwardingBridgeSupport.java:151) > >>>> >>>> > at > >>>> >>>> > > >>>> >>>> > > >>>> > org.apache.activemq.transport.ResponseCorrelator.onCommand(ResponseCorrelator.java:116) > >>>> >>>> > at > >>>> >>>> > > >>>> >>>> > > >>>> > org.apache.activemq.transport.MutexTransport.onCommand(MutexTransport.java:50) > >>>> >>>> > at > >>>> >>>> > > >>>> >>>> > > >>>> > org.apache.activemq.transport.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:113) > >>>> >>>> > at > >>>> >>>> > > >>>> >>>> > > >>>> > org.apache.activemq.transport.AbstractInactivityMonitor.onCommand(AbstractInactivityMonitor.java:229) > >>>> >>>> > at > >>>> >>>> > > >>>> >>>> > > >>>> > org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:83) > >>>> >>>> > at > >>>> >>>> > > >>>> >>>> > > >>>> > org.apache.activemq.transport.tcp.SslTransport.doConsume(SslTransport.java:91) > >>>> >>>> > at > >>>> >>>> > > >>>> >>>> > > >>>> > org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:222) > >>>> >>>> > at > >>>> >>>> > > >>>> > org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:204) > >>>> >>>> > > >>>> >>>> > 2012-05-11 05:24:37,249 | DEBUG | mstmip102.vosh.net adding > >>>> >>>> > destination: > >>>> >>>> > topic://ActiveMQ.Advisory.Connection | > >>>> >>>> > org.apache.activemq.broker.region.AbstractRegion | > StopAsync:vm:// > >>>> >>>> > mstmip102.vosh.net#0-23 > >>>> >>>> > 2012-05-11 05:24:37,250 | WARN | Failed to remove connection > >>>> >>>> > ConnectionInfo {commandId = 3, responseRequired = false, > >>>> connectionId > >>>> >>>> > = > >>>> >>>> > ID:msipap102-50908-1336690011745-7:1, clientId = > >>>> >>>> > mstmip102_msipap102.vosh.net_outbound, clientIp = > >>>> >>>> > vm://mstmip102.vosh.net#0, > >>>> >>>> > userName = mqipauser, password = *****, brokerPath = null, > >>>> >>>> > brokerMasterConnector = false, manageable = false, > clientMaster = > >>>> >>>> > true, > >>>> >>>> > faultTolerant = false, failoverReconnect = false}, reason: > >>>> >>>> > java.lang.IllegalStateException: Timer already cancelled. | > >>>> >>>> > org.apache.activemq.broker.TransportConnection.Service | > >>>> >>>> > StopAsync:vm:// > >>>> >>>> > mstmip102.vosh.net#0-23 > >>>> >>>> > 2012-05-11 05:24:37,250 | DEBUG | Exception detail: | > >>>> >>>> > org.apache.activemq.broker.TransportConnection.Service | > >>>> >>>> > StopAsync:vm:// > >>>> >>>> > mstmip102.vosh.net#0-23 > >>>> >>>> > java.lang.IllegalStateException: Timer already cancelled. > >>>> >>>> > at java.util.Timer.sched(Timer.java:397) > >>>> >>>> > at java.util.Timer.schedule(Timer.java:248) > >>>> >>>> > at > >>>> >>>> > > >>>> >>>> > > >>>> > org.apache.activemq.thread.Scheduler.schedualPeriodically(Scheduler.java:49) > >>>> >>>> > at > org.apache.activemq.broker.region.Topic.start(Topic.java:530) > >>>> >>>> > at > >>>> >>>> > > >>>> >>>> > > >>>> > org.apache.activemq.broker.region.AbstractRegion.addDestination(AbstractRegion.java:143) > >>>> >>>> > at > >>>> >>>> > > >>>> >>>> > > >>>> > org.apache.activemq.broker.region.RegionBroker.addDestination(RegionBroker.java:286) > >>>> >>>> > at > >>>> >>>> > > >>>> >>>> > > >>>> > org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:145) > >>>> >>>> > at > >>>> >>>> > > >>>> >>>> > > >>>> > org.apache.activemq.advisory.AdvisoryBroker.addDestination(AdvisoryBroker.java:161) > >>>> >>>> > at > >>>> >>>> > > >>>> >>>> > > >>>> > org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:145) > >>>> >>>> > at > >>>> >>>> > > >>>> >>>> > > >>>> > org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:145) > >>>> >>>> > at > >>>> >>>> > > >>>> >>>> > > >>>> > org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:145) > >>>> >>>> > at > >>>> >>>> > > >>>> >>>> > > >>>> > org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:145) > >>>> >>>> > at > >>>> >>>> > > >>>> >>>> > > >>>> > org.apache.activemq.security.AuthorizationBroker.addDestination(AuthorizationBroker.java:81) > >>>> >>>> > at > >>>> >>>> > > >>>> >>>> > > >>>> > org.apache.activemq.broker.MutableBrokerFilter.addDestination(MutableBrokerFilter.java:151) > >>>> >>>> > at > >>>> >>>> > > >>>> >>>> > > >>>> > org.apache.activemq.broker.region.RegionBroker.send(RegionBroker.java:481) > >>>> >>>> > at > >>>> >>>> > > >>>> >>>> > > >>>> > org.apache.activemq.broker.jmx.ManagedRegionBroker.send(ManagedRegionBroker.java:305) > >>>> >>>> > at > >>>> >>>> > > >>>> >>>> > > >>>> > org.apache.activemq.advisory.AdvisoryBroker.fireAdvisory(AdvisoryBroker.java:533) > >>>> >>>> > at > >>>> >>>> > > >>>> >>>> > > >>>> > org.apache.activemq.advisory.AdvisoryBroker.fireAdvisory(AdvisoryBroker.java:464) > >>>> >>>> > at > >>>> >>>> > > >>>> >>>> > > >>>> > org.apache.activemq.advisory.AdvisoryBroker.fireAdvisory(AdvisoryBroker.java:459) > >>>> >>>> > at > >>>> >>>> > > >>>> >>>> > > >>>> > org.apache.activemq.advisory.AdvisoryBroker.removeConnection(AdvisoryBroker.java:234) > >>>> >>>> > at > >>>> >>>> > > >>>> >>>> > > >>>> > org.apache.activemq.broker.BrokerFilter.removeConnection(BrokerFilter.java:113) > >>>> >>>> > at > >>>> >>>> > > >>>> >>>> > > >>>> > org.apache.activemq.broker.TransactionBroker.removeConnection(TransactionBroker.java:351) > >>>> >>>> > at > >>>> >>>> > > >>>> >>>> > > >>>> > org.apache.activemq.broker.BrokerFilter.removeConnection(BrokerFilter.java:113) > >>>> >>>> > at > >>>> >>>> > > >>>> >>>> > > >>>> > org.apache.activemq.security.JaasAuthenticationBroker.removeConnection(JaasAuthenticationBroker.java:95) > >>>> >>>> > at > >>>> >>>> > > >>>> >>>> > > >>>> > org.apache.activemq.broker.BrokerFilter.removeConnection(BrokerFilter.java:113) > >>>> >>>> > at > >>>> >>>> > > >>>> >>>> > > >>>> > org.apache.activemq.broker.MutableBrokerFilter.removeConnection(MutableBrokerFilter.java:119) > >>>> >>>> > at > >>>> >>>> > > >>>> >>>> > > >>>> > org.apache.activemq.broker.TransportConnection.processRemoveConnection(TransportConnection.java:766) > >>>> >>>> > at > >>>> >>>> > > >>>> >>>> > > >>>> > org.apache.activemq.broker.TransportConnection.doStop(TransportConnection.java:1068) > >>>> >>>> > at > >>>> >>>> > > >>>> >>>> > > >>>> > org.apache.activemq.broker.TransportConnection$4.run(TransportConnection.java:996) > >>>> >>>> > at java.lang.Thread.run(Thread.java:722) > >>>> >>>> > > >>>> >>>> > 2012-05-11 05:24:37,252 | DEBUG | mstmip102.vosh.net adding > >>>> >>>> > destination: > >>>> >>>> > temp-queue://ID:POTMIP102-54337-634722930065323786-1:18:1 | > >>>> >>>> > org.apache.activemq.broker.region.AbstractRegion | ActiveMQ > >>>> Transport: > >>>> >>>> > ssl:///172.17.5.78:59658 > >>>> >>>> > > >>>> >>>> > 2012-05-11 05:24:37,341 | DEBUG | bridging (mstmip102.vosh.net-> > >>>> >>>> > msipap101.vosh.net) > >>>> ID:MSTMIP102-59588-1336713700699-1:1:0:0:25184, > >>>> >>>> > consumer: ID:MSTMIP102-59588-1336713700699-4:2:1:1, destination > >>>> >>>> > topic://ActiveMQ.Advisory.TempQueue, brokerPath: > >>>> >>>> > [ID:MSTMIP102-59588-1336713700699-0:1], message: > ActiveMQMessage > >>>> >>>> > {commandId > >>>> >>>> > = 0, responseRequired = false, messageId = > >>>> >>>> > ID:MSTMIP102-59588-1336713700699-1:1:0:0:25184, > >>>> originalDestination = > >>>> >>>> > null, > >>>> >>>> > originalTransactionId = null, producerId = > >>>> >>>> > ID:MSTMIP102-59588-1336713700699-4:1:1:1, destination = > >>>> >>>> > topic://ActiveMQ.Advisory.TempQueue, transactionId = null, > >>>> expiration > >>>> >>>> > = 0, > >>>> >>>> > timestamp = 0, arrival = 0, brokerInTime = 1336713871588, > >>>> >>>> > brokerOutTime = > >>>> >>>> > 1336713877341, correlationId = null, replyTo = null, > persistent = > >>>> >>>> > false, > >>>> >>>> > type = Advisory, priority = 0, groupID = null, groupSequence = > 0, > >>>> >>>> > targetConsumerId = null, compressed = false, userID = null, > >>>> content = > >>>> >>>> > null, > >>>> >>>> > marshalledProperties = > >>>> org.apache.activemq.util.ByteSequence@567edec8, > >>>> >>>> > dataStructure = DestinationInfo {commandId = 0, > responseRequired = > >>>> >>>> > false, > >>>> >>>> > connectionId = ID:MSTMIP102-59588-1336713700699-3:2, > destination = > >>>> >>>> > temp-queue://ID:POTMIP102-54337-634722930065323786-1:18:1, > >>>> >>>> > operationType = > >>>> >>>> > 0, timeout = 0, brokerPath = null}, redeliveryCounter = 0, > size = > >>>> >>>> > 1024, > >>>> >>>> > properties = {originBrokerURL=ssl://MSTMIP102:61616, > >>>> >>>> > originBrokerId=ID:MSTMIP102-59588-1336713700699-0:1, > >>>> originBrokerName= > >>>> >>>> > mstmip102.vosh.net}, readOnlyProperties = false, readOnlyBody > = > >>>> false, > >>>> >>>> > droppable = false} | > >>>> >>>> > org.apache.activemq.network.DemandForwardingBridgeSupport | > >>>> ActiveMQ > >>>> >>>> > Connection Dispatcher: vm://mstmip102.vosh.net#4 > >>>> >>>> > > >>>> >>>> > 2012-05-11 05:24:37,341 | DEBUG | bridging (mstmip102.vosh.net-> > >>>> >>>> > msipap101.vosh.net) > >>>> ID:MSTMIP102-59588-1336713700699-1:1:0:0:25185, > >>>> >>>> > consumer: ID:MSTMIP102-59588-1336713700699-4:2:1:1, destination > >>>> >>>> > topic://ActiveMQ.Advisory.TempQueue, brokerPath: > >>>> >>>> > [ID:MSTMIP102-59588-1336713700699-0:1], message: > ActiveMQMessage > >>>> >>>> > {commandId > >>>> >>>> > = 0, responseRequired = false, messageId = > >>>> >>>> > ID:MSTMIP102-59588-1336713700699-1:1:0:0:25185, > >>>> originalDestination = > >>>> >>>> > null, > >>>> >>>> > originalTransactionId = null, producerId = > >>>> >>>> > ID:MSTMIP102-59588-1336713700699-4:1:1:1, destination = > >>>> >>>> > topic://ActiveMQ.Advisory.TempQueue, transactionId = null, > >>>> expiration > >>>> >>>> > = 0, > >>>> >>>> > timestamp = 0, arrival = 0, brokerInTime = 1336713871588, > >>>> >>>> > brokerOutTime = > >>>> >>>> > 1336713877341, correlationId = null, replyTo = null, > persistent = > >>>> >>>> > false, > >>>> >>>> > type = Advisory, priority = 0, groupID = null, groupSequence = > 0, > >>>> >>>> > targetConsumerId = null, compressed = false, userID = null, > >>>> content = > >>>> >>>> > null, > >>>> >>>> > marshalledProperties = > >>>> org.apache.activemq.util.ByteSequence@235448d5, > >>>> >>>> > dataStructure = DestinationInfo {commandId = 0, > responseRequired = > >>>> >>>> > false, > >>>> >>>> > connectionId = ID:MSTMIP102-59588-1336713700699-3:2, > destination = > >>>> >>>> > temp-queue://ID:POTMIP102-54337-634722930065323786-1:18:1, > >>>> >>>> > operationType = > >>>> >>>> > 1, timeout = 0, brokerPath = null}, redeliveryCounter = 0, > size = > >>>> >>>> > 1024, > >>>> >>>> > properties = {originBrokerURL=ssl://MSTMIP102:61616, > >>>> >>>> > originBrokerId=ID:MSTMIP102-59588-1336713700699-0:1, > >>>> originBrokerName= > >>>> >>>> > mstmip102.vosh.net}, readOnlyProperties = false, readOnlyBody > = > >>>> false, > >>>> >>>> > droppable = false} | > >>>> >>>> > org.apache.activemq.network.DemandForwardingBridgeSupport | > >>>> ActiveMQ > >>>> >>>> > Connection Dispatcher: vm://mstmip102.vosh.net#4 > >>>> >>>> > > >>>> >>>> > 2012-05-11 13:29:40,053 | WARN | TopicSubscription: > >>>> >>>> > consumer=ID:MSTMIP102-32947-1336741838889-4:2:1:1, > destinations=7, > >>>> >>>> > dispatched=1000, delivered=7723, matched=1001, discarded=0: has > >>>> twice > >>>> >>>> > its > >>>> >>>> > prefetch limit pending, without an ack; it appears to be slow | > >>>> >>>> > org.apache.activemq.broker.region.TopicSubscription | ActiveMQ > >>>> >>>> > Transport: > >>>> >>>> > ssl:///10.60.252.28:54614 > >>>> >>>> > > >>>> >>>> > 2012-05-11 13:29:42,540 | WARN | TopicSubscription: > >>>> >>>> > consumer=ID:MSTMIP102-32947-1336741838889-3:2:1:1, > destinations=6, > >>>> >>>> > dispatched=1000, delivered=9922, matched=1001, discarded=0: has > >>>> twice > >>>> >>>> > its > >>>> >>>> > prefetch limit pending, without an ack; it appears to be slow | > >>>> >>>> > org.apache.activemq.broker.region.TopicSubscription | ActiveMQ > >>>> >>>> > Transport: > >>>> >>>> > ssl:///10.60.252.28:54614 > >>>> >>>> > >>>> >>>> > >>>> >>>> > >>>> >>>> -- > >>>> >>>> http://fusesource.com > >>>> >>>> http://blog.garytully.com > >>>> >>> > >>>> >>> > >>>> >> > >>>> > > >>>> > >>>> > >>>> > >>>> -- > >>>> http://fusesource.com > >>>> http://blog.garytully.com > >>>> > >>> > >>> > >> > > > > -- > http://fusesource.com > http://blog.garytully.com >
